Yeah, I reflected on your comment for a bit and I can see what you’re saying. All software uses dependencies, but how they are shipped is different for a lot of languages. If we take an extreme example, Go has a lot of the same microdependencies that Python, JS, etc. have, (probably a lot more than Python even), but since programs are usually compiled into a standalone binary, you never think about them as a user. Interpreted languages on the other hand tend to not do this because they’re not compiled, and when you’re installing a dependency, you’re often installing the source code itself, rather than a shared library or .jar like you would in C++ or Java.

A lot of people do complain about virtual environments though, and while I don’t personally have a problem with them, I do wonder why nobody uses bundlers in Python. It seems that something like this already exists, but it doesn’t seem to be widely adopted or used for all the complaints I hear about virtual environments and such.

It’s not a question of what other browsers would offer, that doesn’t really matter all that much. From Mozilla’s perspective, it has more to do with the fact that Firefox has to keep up with a behemoth that has several times the amount of money and resources that they have.

But for you and me, it’s something completely different, and much more ominous than whatever Mozilla’s issues are.

The truth is that, now that Chrome has monopolistic control over the web, they can make it into whatever they want it to be. And I frankly don’t want to find out what that is. Maybe it’s the return of AMP, maybe it’s more invasive DRM, maybe it’s forcing you to put your ID into the browser so they know who you are and can blacklist you from the web if they have a vendetta against you. Maybe it’s the fact that they won’t let you browse the web on a computer if it doesn’t have a locked bootloader and rootless access to the OS the same way Android does.

These may sound like crazy ideas today, but bear in mind that the hell we’re living today is the long term realization of the crazy ideas of a few decades ago. Why should we want to see what that’s like? Why shouldn’t we be trying whatever we can do to prevent it?

Funny you mention that: https://www.panamericandispatch.org/p/meta-ignores-brazilian-court-order

Really hope Facebook gets pounded for their interference in Brazil’s election.

Google has done more for web standards than the open source community ever could

Google was a great company when they were an upstart that needed to create standards so they could compete with the likes of Microsoft and Apple. They pushed for standards, adopted them, and promoted others to do the same.

Today, they have weaponized standards to entrench their monopolistic positions. They pushed so many standards in CSS, Javascript, and other technologies that it’s impossible for a competing browser to emerge outside of Chrome, which they own, and Firefox, which… they also own in any way that matters. And they also leveraged their search monopoly to favor websites that adopted the standards they were trying to push in order to force the web to adopt the standards they wrote, and therefore, their own browser.

They helped create the open web. Then they claimed it as their own.

It’s a great example of how a group starts out being your friend, but then amasses all the power for itself and gets corrupted by it. They have not been our friends for over a decade now, and we can all see how bad it gets.

Unit tests are good IMO, but integration tests are also important. Unit tests are too discrete and don’t expose system wide issues that cross the boundaries of what you’re testing at the unit level. Examples: performance, stability, memory leaks, bugs that are caused by how multiple components interact with each other, etc.

The other thing is that I think people use tests to give a false sense of security. Tests cannot expose issues that aren’t explicitly covered by testing. Design validation is still important and people tend to overlook that because of the false sense of security they get from seeing a test suite pass.

Pretty much all software uses dependencies, my friend.

https://github.com/kubernetes/kubernetes/blob/release-1.23/go.mod

The Linux Foundation has unfortunately become a commercial removed, run by a clusterfuck of Silicon Valley VCs. When I see the kinds of garbage AI projects that are being shoveled into it every day, I lose more and more respect for it.

Look for yourselves: https://www.linuxfoundation.org/projects

And it’s sad to see Linus falling into their lock and step. I respect him for a lot of things, but when it comes to AI, he’s sadly just wrong.

Computer kids will always be more tech savvy than touchscreen kids. They’ll have better critical thinking skills and better abilities to apply their knowledge in ways that aren’t spelled out for them by patronizing tech companies like Google and Apple pushing their walled gardens.

Great news, not just for Brazil, but for the whole world.

But winning the election is just the first step. The more important part is that he uses his victory to deliver the results that he promised. If politicians did this better, we would never see a far right movement in the first place.

Let’s hope we see it happen.

If you didn’t automatically register that requiring spawning full OS processes to scale performance immediately disqualifies a language from a whole plethora of use-cases

That’s way too broad of a statement to be true in any absolute sense. It really depends on what you’re doing, and sure, there are plenty of ways to do it stupidly, but a lot of people that are smarter than you and me have thought about ways that the overhead of spawning processes can be mitigated. All I can say is that I’ve had perfectly good success using multiprocessing.Pool and getting near perfect speedups. Is it as fast as threading? No. But it’s good enough. If it’s not good enough for you, then maybe we just have different use-cases, but without talking about concrete examples, we can’t make any specific claims.

And we didn’t even get to low level stuff yet. People are already failing at the macro level. Sad.

What do you want to talk about? Are you writing kernel drivers in Python or something? Why would we talk about writing low level code in Python, that sounds absurdly stupid. Most of the high performance libraries people use are written in Fortran or C. The entire field of data science uses Python, if performance were not a heavily optimized problem, then a lot of people out there must really love wasting their money on compute. Python is a glue language for high performance low level code, everybody knows this already.

Frankly this thread has been very illuminating. My perception of the average p.d user and their concrete knowledge was very inflated.

At least we agree on something!

If you didn’t use a profiler, then how can you be sure that the issue was Python and not something else? If the slowness really is evenly distributed through the code as you suggest, then a profiler would show that almost instantly and it’s very easy to do that in Python using cProfile.

On top of that, the Python standard library is built for convenience, not performance. The nature of it being a standard library means that the implementation has to be simple. fastregex uses LLVM and SIMD for example, that’s something that’s wildly impractical for the Python stdlib to support. In contrast, I have no idea what regex or JSON library you used for your Rust code, but if it were outside the stdlib, chances are high that it had much more leeway to optimize performance because it didn’t have all the other maintenance obligations that stdlib code needs to support.

All of this is just pure speculation on my part, but ultimately I’m trying to make a point that it is hard to say that the difference was caused by a language issue and not something else. There are lots of other things that could be contributing to this difference and you can’t really say for sure without meaningful diagnostics.

As an example of this, look at Nicholas Nethercote’s excellent series of blogs on making the Rust compiler faster: https://nnethercote.github.io/

The “bottlenecks” they find result in 1% speedups.

The kinds of software that you and I write on a daily basis is nothing compared to pieces of software that get millions of man-hours per year. Assuming you’re not Linus Torvalds or Ted Ts’o hiding behind a funny alias, of course. I don’t think anyone is suggesting that it’s a good idea to write a compiler in Python (outside of prototyping purposes perhaps) since it’s obviously not I/O bound, but if you and I are trying to optimize something we wrote ourselves, we’re more likely to be in the “nice small hotspot” domain you described, rather than the 1% bottleneck speedups discussed in the link you posted.

Very cool link though by the way, I look forward to reading as much of it as I can…

Ah okay, I see what you’re saying. GIL removal can break FFI code through the C API.

I have a lot of thoughts on this, and most of it is too incoherent to put into a comment someone else can understand, but for the most part, I think the issues are still pretty overblown. I just can’t see a world where Python needs true multithreading support when multiprocessing works just fine for most applications. You find a hot loop, you parallelize it in multiprocessing and just like that, your process runs much faster.

I do think that GIL removal is a terrible idea in general, for the reasons in the quote you provided, but at least it’s optional. I would personally not enable it in anything I maintain for several years if I didn’t have to.

I also think that polyglot code running in the same process is also a bad idea. I can understand calling C or Fortran from Python for numerical processing for example, but doing the reverse (like calling Python from Rust) is something I try to avoid at all costs. In any project I’ve worked on, if significant portions of the codebase were written in different languages, you would just use something like protobuf to specify the I/O protocol between the different languages and call them across a socket. It’s easier to maintain in my opinion since it defines a concrete boundary between the two different portions of code, and also makes it much easier to test each component separately too.

Sure, it’s slow. Google’s landing page is 4 MB for a small graphic, a text form, and some buttons. But what are you going to do? Not use it? They won. They don’t have to fix shit.

This neglects to include the massive amount of additional front-end code Google (and other large tech companies) toss on their webpages for things that are purely for their own benefit while being of zero or negative value to the users. Mouse tracking, analytics, fingerprinting, and all kinds of other harmful content that tech companies put out go a long way towards reaching that 4MB figure.

It’s not so much “But that’s not how you build stable software.” It’s more like “But that’s not how you build software [at all today].” Modern software is powered by loads of shared packages, that’s just how all modern languages work. This is equally true for Rust and Python.

You’re not wrong - all libraries with automated dependency packaging do have this problem. But you’re downplaying the scale by a lot. Javascript dependency graphs are absolutely massive. Python has a decent standard library that reduces the number of shared packages by quite a bit. There are definitely downsides to having a large standard library like Python has, but it does significantly reduce the probability of a supply chain attack compared to Javascript.

I can’t comment on Rust personally, since I don’t have much experience with the ecosystem.

Or the fact that you can’t do concurrency especially across the language barrier from another language without multiple python runtimes orchestrated with a queue and IPC because of possible process-wide global state and similar concerns/assumptions for any python library dependency that is not all native across its own dependency chain?

This is very out of date. I haven’t written Python for over 4 years and even back then, I was using multiprocessing.Pool for pretty much all parallel processing that I ever needed. Sure, it’s doesn’t fit every use case for concurrency, but for simple stuff it’s perfectly fine.

Not only that, it’s one of the simplest parallel processing APIs I’ve ever used:

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

Did you use a profiler to find the bottlenecks? Did you try using cjson to speed up the json parsing? I would also avoid using the stdlib regex library too. It’s probably more of a convenience/reference implementation, not something to be intended for high performance. Maybe there’s a re2 wrapper for Python or a similar regex library in C with Python bindings.

Rewriting in Rust is fine, but good Python devs know how to find bottlenecks in their software and replace them with fast implementations. I’ve written plenty of performant Python code, it’s not very hard to profile and optimize. Usually the performant stuff is already written in C/C++ or Fortran anyway (numpy, PyQt, etc.)

Just be grateful that you don’t have to pay for the privilege of watching ads… yet.

Cable TV, I’m looking at you!

I read a hilarious article, probably 2 decades ago by now, where a guy decided to try running Windows viruses on Linux in Wine.

If this really is a threat today, I would say that Wine has become a pretty damn impressive project. Here’s the link in case anyone else wants to read it: https://entertainment.slashdot.org/story/05/01/26/219201/running-windows-viruses-under-linux

The end may be peaceful, but he didn’t say how rough the journey could be!

wtf are you torrenting now, coffee pot???