Hacker News new | past | comments | ask | show | jobs | submit login
Erlang: The coding language that finance forgot (2022) (efinancialcareers.com)
182 points by mindcrime on Feb 2, 2023 | hide | past | favorite | 152 comments



The reason Erlang (and Elixir) are interesting are not the languages themselves (although Elixir is a very nice language in my opinion), but the runtime behind them, BEAM.

The BEAM is the brainchild of the excellent engineers at Ericsson, the product of decades of R&D into building performant, scalable and reliable software for their telephony products.

It scales like a dream, runs your application on all available cores with no trouble at all, handles massive amounts of concurrent requests without breaking a sweat. The programming techniques it enables with projects like Phoenix LiveView are unique and powerful.


What's the license for the BEAM runtime? I've found that Erland and Elixir are both Apache License, but I can't find out whether the bytecode they generate can be executed in a fully open source environment.


AFAIK, BEAM is developed within the erlang/otp, so the license is Apache License 2.0.

Not sure why bytecode generated by a APLv2 project couldn't be executed in a fully open source environment.


I have no idea either but it has a whiff of Oracle to it.



Does "BEAM" mean anything / is an acronym?


Originally “Bogdan's Erlang Abstract Machine”, but then the subsequent maintainer was named Björn so you could sub that in too.


IIRC Joe Armstrong said he was puzzled why people latched on to the term "BEAM" so much, and he preferred that it was simply referred to as the Erlang VM.


Beam, with one syllable, is more convenient to say out loud than Erlang VM. It also sounds cooler.


In fact, it sounds awesome!


And the predecessor to BEAM was JAM (Joe’s) which is a bit of a shame we didn’t end up with that acronym… or not, I guess.


While the BEAM is the real magic, the languages (Erlang and later Elixir) are designed specifically to enable that. You can't just decouple the two.


Erlang isn't dying but not many people care for it in capital market. Looks at the traits of Erlang, it's designed for telecom environment which is almost opposite of capital market.

1. Fault tolerance is almost all hardware driven. Capital market participants spent billions of dollar building reliable private networks.

2. Distributed in trading doesn't mean what FAANG are doing. Trading systems for example usually run 2 hot-warm systems with real-time replication commits, that's about it. They don't run horizontally scaled systems because network time would kill their performance.

In my experience, I haven't heard anyone asking for hot swap code and high availability (outside of trading hours).

Are there shops out there that run Erlang? Sure, but most companies don't care about Erlang simply because its value isn't better than Java/C/C++.


Funny anecdote about one of the FAANGs: before the idea of service meshes and Consul and Envoy a similar idea was being rolled out to transparently proxy, encrypt, and authenticate traffic between services. The initial implementation was on erlang and didn’t cause any operational issues for our team, from what I remember. The project was moved to another team who decided for version 2 to rewrite in Java (high made sense from an integration perspective).

Unfortunately, they hadn’t tested at scale along with whatever it was servicing and as it was being rolled out it quickly ballooned in memory causing systems to swap and fall over. They eventually got it working, but the rollout was less than ideal and unfortunately an erlang project that affected a significant amount of traffic without notice was deprecated.


Everyone always talks/jokes about, "Rewrite it in Rust," think about all the business applications over the years that had been rewritten to Java, just because. Usually under the guise of, "Kids coming out of college will have an easier time wrapping their heads around Java," than some other lang.

In my time as an engineer, I've seen quite a few Scala and Clojure apps, get rewritten in this manner.

This is all antidotal, of course.


I sometimes wonder if this logic makes sense. I'm currently leading a fully inexperienced team where my main challenge isn't delivering products but qualified people. I have to make them learn so many company specific and codebase specific things that feels like learning a new language would be a least of it. This is of course colored by my language learning loving lens, so I can't be sure about this.


I've been on teams responsible for components written in more niche, powerful languages by previous employees who were deep in that particular language community. The author was really smart, the code was good. It was a big headache. First off, different build setup, different runtime, couldn't use most of our standard libs, so it was an island. It being in a language most people didn't know well meant it mostly only got updated as necessary; we certainly had people who could read the code and make tweaks with some comfort, but it was mostly just tweaks because nobody had the depth to really engage with the high level design idioms. It being a niche lang/community, it moved fast, so after a year or so our tool was apparently using deprecated practices; in our supported languages, teams did large-scale fixup across repos for new versions, but this being a special lang/toolchain, it was on us. Eventually, I believe it got rewritten, somewhat simpler, in one of our supported languages, and it was fine.

Maybe this is a success story: someone made a useful tool in a weird language and we used it successfully for at least a year. Maybe it wouldn't have been made or would've been worse or more expensive if they'd been forced to write it in a supported language. Maybe it would've needed to be rewritten anyway.

My view is a simple one; as one of the leads responsible for it, it was a special case bit of code, something people tried to avoid touching. Nothing about it required a niche language technically, so the added difficulty of working around that was friction and risk. It wasn't that people couldn't learn the language; they could, we had lang PhDs. It's just better if you can keep people where they have existing expertise, and spend your learning points on solving problems users care about.


I don't think "avoid otherwise-unused languages for secondary tools" is the same as "avoid less common languages for your main product". Every software developer at your company will likely work on your main product for a non-trivial amount of time. Any software developer who can't figure out a new language in a trivial amount of time isn't worth hiring. Thus, everyone you hire will eventually be competent in the language of your main product.

In contrast, for secondary products, there's a good chance most developers will never work on it at all, and of those who do a significant fraction will spend only a trivial amount of time. Adding the overhead of working in a new language to that balance is significant.

Therefore, the ideal is "Choose the best language for your main product, then write all supporting software either in that language, or a widely-used lingua franca like Python".


Porting Clojure and Scala to Java can also have performance advantages, but yeah, Java was an industrial language before Golang made it cool.


I’m running into this concern at work. If you have one main app or service constellation, then you can’t make any big changes like these because they have to scale from test benchtop to production with few steps in between and there’s just no accurate way to know what you’re going to see until you try. Traffic shaping only does so much.

I’ve been moving our tools and sidecars and lesser services to do more reuse, not for the code and effort duplication reasons, but so these technologies get more burnin time before they have to do it live. We have one service that is particularly good for this because it doesn’t see a lot of traffic on average, but it bursts up to production requests levels during a certain workflow that is less time sensitive than live traffic but still gets scrutiny because it blocks a run book until it’s done. Parts of that run book can be run as often as you like if you stop before the steps that alter production state, so you can run it a few times to take the median time and see if you’ve improved or regressed.

We turn things on there and if they haven’t caught on fire we feel a lot better about flipping it on for user facing traffic.


The company I currently work for switched from Java to Elixir/Erlang because developer productivity was so much higher. Being able to turn around features and fixes in a fraction of the time has been quite valuable for us.


I would love to hear more about this if you're willing! Java shops (In my experience) have been the most unlikely to switch shops there are. Some questions:

What pain were you feeling?

Who came up with the idea/proposal to switch and how did they deliver it?

Was it hard to get people on board?

How difficult was it for people to learn Elixir?

Did people take to it quickly and/or were there people who had a hard time?

How long did the transition take to complete?


Hello, freedomben. I'm back from my business trip, so here are my answers to your questions, as promised.

> What pain were you feeling?

The turn-around time for refactoring, adding new features, changing API endpoints, etc. was taking too long in Java.

> Who came up with the idea/proposal to switch and how did they deliver it?

The engineering team, staff engineers, and the VP of Engineering recognized the productivity issues and decided to experiment with other languages to see if a switch would result in quicker turn-around times.

First, we came up with a long list of objective metrics we could use to test various languages against each other.

Next, we made a list of languages we wanted to test (e.g., Java, Elixir, Go, Clojure, etc.).

Then, we defined a test program, including a REST API, that we would build to compare the languages.

Finally, we broke up into small teams, assigned each team a language to test, and spent about a week writing our test program in a variety of languages.

Once we were done, we had a retrospective and scored our experiences with each language we tested using the list of objective metrics we had created.

In the end, Elixir won.

> Was it hard to get people on board?

It wasn't hard to convince the team we should switch to Elixir. The team's own test data revealed that.

> How difficult was it for people to learn Elixir?

It wasn't too bad. Most of our engineers grasped it quickly. I think two factors made the switch successful:

1. Two of us already had professional Elixir experience and could mentor other team members.

2. The company purchased team licenses for the Pragmatic Studio's Elixir courses, which were invaluable.

> Did people take to it quickly and/or were there people who had a hard time?

Most of our engineers were able to pick up Elixir quickly. Some of the backend data engineers took longer though, just because they didn't have to work with it every day.

> How long did the transition take to complete?

We completed the rewrites of all our products in less than a year.

I hope this was useful. Let me know if you have any other questions.


I’m happy to share, but I’m traveling and only have my phone. I’ll type it up when I get home this weekend.


What industry/domain is the software you write in?


Data-driven insights.


> 2. Distributed in trading doesn't mean what FAANG are doing. Trading systems for example usually run 2 hot-warm systems with real-time replication commits, that's about it. They don't run horizontally scaled systems because network time would kill their performance.

That's pretty close to Ericsson's use case for dist; they built it to run redundant computers in the same chassis for telephone switches. It just also happens to work fairly well at much larger cluster sizes too.


Finance means different things to different people. Are we talking about trading "finance" (including high frequency trading), investment banking "spreadsheet" finance, consumer or corporate "database" finance or maybe even (the horror, the horror) "defi" cryptofinance?

What is true is that finance in almost all its guises is a world apart in terms of information technology. Closed and proprietary, ultra expensive and arcane, highly regulated etc., it is an information technology branch that evolved separately and it shows.

Whether that will remain so is an interesting question. So called "fintech" has been a thing for a while. E.g. whatsapp is famously built on erlang, so if Meta wants to turn it into a super-app (the "Asian way") you will have a de-facto application in "consumer" mobile/payments finance.


The one big use case was RabbitMQ in a messaging app, not HFT. I doubt Elixir even with Nx can compete with low-level HFT code. Python DL/ML code libraries are just wrappers around C too. Maybe if BeamAsm and Nx are used Elixir could be used for more numerical or not just distributed applications.

I've programmed in Python and Julia, and when I worked at an engineering (mechanical, entertainment engineering) company, Julia was great for its similarity to Matlab. I am a self-taught engineer, so I did not get pulled into Matlab in college.

Personally, I took to Erlang, so I could write plugins for Wings3D back in the early 2000s, but I never stuck with Erlang, or Wings3D (Blender3D was my choice and I even contributed to have it go opensource way back when). I like Erlang's syntax better for some reason, although Elixir's is beautiful too. I was not a Ruby programmer, and I had delved into Haskell and Prolog, so I think Erlang made more sense to me. I think Elixir has a lot more momentum behind it than Erlang, but at the root it's Erlang, so I think I'll stick with Erlang for BEAM apps. My favorite language is April[1] (APL in Lisp), and given my love of J, would be a better fit for any finance apps I might write. I am trying to convert some of the Lisp code in this book, "Professional Automated Trading: Theory and Practice" to April.

Maybe I'll write some equivalent Elixir code to compare.

[1] https://github.com/phantomics/april


Given your love for Lisp, and your knowledge of Erlang, perhaps Robert Virding's Lisp Flavored Erlang (LFE) might be of interest?

http://lfe.io -- among other things, the site has tutorials and some edits of Lisp books (e.g. SICP) rewritten to use LFE.

(And, from what I've heard, LFE is being used in financial apps, though perhaps more along the lines of expert-systems than regular financial apps? (Sadly, my knowledge is at best second-hand)).

P.S. Nice to finally run into someone who has actually used Wings3D -- it gets brought up as an example of a significant Erlang project, and I know there's a community of people who were/are using it, but, since it's not a technically-oriented tool, even at Erlang conferences one is unlikely to run into Wings3D people.


I delved into LFE, and I was a proponent when Elixir was just catching on, but Elixir's momentum just washed it away. I believe Elixir earned it, but I think it had a heavy kickstart from all of the Ruby programmers who took to it. And they say syntax doesn't matter ;)

It is my understanding LFE makes a lot of compromises as a list in trying to work nicely with the Erlang VM (BEAM). I really wish Gleam kept the more Haskelly/ML syntax instead of shooting for the popularity contest by becoming more Algol or C-like. Slim Whitman allegedly sold more records than the Beatles, but I would rather listen to the Beatles!


Wasn't WeChat built in Erlang as well?


I think WhatsApp was Elixir


I think WhatsApp was Elixir

Erlang, actually; and they're still using it -- here's a paper-abstract[0] from Sept.2022 (and video of a talk[1]) about their open-source "EqWAlizer" tool[2] for statically-typing Erlang (as-is).

(EqWAlizer was discussed on an Erlang-related podcast[3], including its relationship to Dialyzer -- the original Erlang type-checking tool -- and it was briefly discussed on HN[4]).

[0] https://dl.acm.org/doi/10.1145/3546186.3552537

[1] https://www.youtube.com/watch?v=do9f2FKsKxM

[2] https://github.com/WhatsApp/eqwalizer

[3] https://www.beamrad.io/35

[4] https://news.ycombinator.com/item?id=32330284


Thank you!


Maybe the title would have been better like this: "PROLOG: The coding language that finance forgot."

Erlang (the language) is rooted in Prolog, and many of its best traits can be traced back to this fact. That everything is a pattern, and you can pattern match pretty much anything is rarely found in other languages, for example. Elixir made a mistake in my opinion when it went with a Ruby-like syntax.

Also, there are those kinds of problems where in Prolog you can express and solve them in a few lines, in other languages you struggle a lot (and at the end you just reimplement half of the backtracing engine anyway - see Greenspun's tenth rule). Why it is not more commonly used in banking / finance is one of the greatest mysteries of life.


I think the thing to remember that it all started as an interpreter written in Prolog in which we could develop our ideas on what the real problem was and the right semantics of a system for solving them. As we went along our "language" evolved as well and became less and less Prolog and more functional as we removed much Prolog semantics, added functional "stuff" and developed the final syntax.

We had along the way also looked at concurrent logic languages. So by the time we had a language and design rules how to use it most of Prolog had disappeared, though some of its syntax still remained. This language was, of course, Erlang.

While Prolog was a nice base on which to develop our ideas it was never the language we would have used in real life.


It’s really delightful that we’re able to get the thoughts of someone who was there and involved in the genesis of the language.

Thank you for all the time you’ve taken to explain the “why” of Erlang!

(For those unfamiliar, go check out “The Erlang Rationale” and some of rvirding’s posts elsewhere, e.g., on the thread at https://elixirforum.com/t/the-erlang-rationale-by-robert-vir...)


If you want more, from someone else who was there, check out Joe Armstrong's "History of Erlang" paper:

https://dl.acm.org/doi/abs/10.1145/1238844.1238850 -- canonical link and has a video of the related talk (streamable and downloadable).

(Sadly, without ACM membership, the paper itself is paywalled; there are copies of it online though, e.g. https://www.labouseur.com/courses/erlang/history-of-erlang-a... ).


Erlang as a programming language for writing fault-tolerant systems has quite little in common with Prolog, except the syntax. That is, the semantics of Erlang vastly differ from Prolog.


Elixirs syntax is only Ruby like in that it’s very readable IMO. Any similarity with Ruby stops there.


Well sure but you could be more clear.

Block parameters, implicit parentheses for certain method/function calls, uppercase significance, symbolic significance for brackets.


Elixir doesn't have blocks as a separate concept like Ruby. The "blocks" are just syntax sugar for passing keyword lists of code to macros.

What do you mean by "symbolic significance for brackets?"


I’m not sure the word for it, but brackets only ever mean two things, to my knowledge. Hash or Block, in Ruby.

I spent many years in perl, so I’m of the opinion visual indicators for datatypes are actually a good thing at the point of instantiation/composition.

Not sure how to properly word it


Oh, are all the paren/bracket types interchangeable in Perl? I've never used Perl.

Yea, each set of them ([], (), {}, <>) has a very distinct meaning in Elixir.


Well in perl there’s a concept of sigils.

my @num = (1, 2, 3);

my $num = 25;

say $num[1];

So that last line prints 2, which is obvious by looking at it, but can be difficult in more complex code when you’re overloading names by the way the variable is accessed.

Everything in perl is a rabbit hole of details, as an aside.

Anyhow, I just appreciate that the symbols always mean the same thing. Takes a load off of my brain. Like a single character type annotation, kinda.


backtracking isn't hard to implement, though. Depth-first search is more or less what it is. But erlang lost most of the prolog ancestry, backtracking isn't there, for example. BEAM has a lot of concurrency neatness though, hard to reimplement or even mimic in languages not specifically tailored for from ground up.


I think Elixir's Ruby-like syntax is one of the many things that make it great.

Comparing the two, I think Prolog may be easier for handling databases (I like Ecto, but it can be a little complex to set up and get right). Both Elixir and Prolog handle pattern-matching, but I think Elixir's syntax is nicer to work with. Of course, this is subjective. Both are great at handling list data. IO is consistently easy with Elixir whereas it can be difficult at times with Prolog.

Elixir and Erlang allow you to create supervision trees which can watch code and automatically and quickly recover from unexpected errors. Prolog's error handling and recovery can be slow. The overheard of calling a goal through catch/3, for instance, is quite significant compared to Elixir/Erlang.


Have you seen gleam.run? It’s pretty neat.

I preferred the alpha syntax, which was ML-inspired, but it’s decent in its current incarnation.

I’ve always though the beam was really nice for building programming languages, and clearly that is true, as there are now more than a dozen targeting it.


Thank you for this! I love elixir and Erlang but I hate elixir’s syntax, so Gleam looks really cool!

Are you aware of other beam-targeting languages?


Have I got a link for you!

https://github.com/llaisdy/beam_languages

See you down the rabbit hole!


I will put in a good word for PureScript for the beam with `purerl`. It's my go-to for writing BEAM code nowadays. Notably PureScript tooling including LSP, package management, etc., just works, so you are able to just get to work in internalizing the way OTP and other Erlangy things are expressed in a statically typed, pure language with much better facilities for functional code than any other BEAM language.

https://github.com/purerl/purerl & https://purerl-cookbook.readthedocs.io/ for more information. Join the PureScript discord and the #purerl channel if you want help.


What sort of things do you write with it? You don’t use it professionally do you?


I'm writing proof-of-concept applications in it for our company currently. Our core technology is Erlang (and Elixir), so the jump isn't comparatively as ridiculous as a Java shop suddenly writing PureScript on the BEAM. I trust code written in PureScript far more than I do code written in Elixir and there are already at least 2 companies using `purerl` in production with the majority of their code in PureScript.

With that said, I've been writing Elixir since 2015 and have worked with Erlang & Elixir since a couple of years after that (+ Haskell), so I'm both much more likely to use niche languages and I'm also well versed in using the BEAM. PureScript with `purerl` actually matches Erlang much better than Elixir does, since Elixir spends half the language and standard library trying to not be Erlang, so I find that it's a better match than Elixir even as a citizen of the BEAM, and the type safety is something I've wanted on the BEAM since I first wrote production BEAM code.

All in all I've been blown away by how solid `purerl` is. The runtime is Erlang, the language is excellent; it's what I dreamed about Haskell being but couldn't quite get.


What's the mistake in going with a Ruby-like syntax? I don't necessarily disagree in terms of wider adoption (though I personally really really like the syntax) though the context is unclear around your statement.


No one can clarify? Did I get downvoted for saying I like Ruby syntax or that I didn't understand the context? I don't understand how pattern matching and Ruby syntax have any correlation. I've never heard of any pattern matching possible in Erlang that was made impossible in Eilxir due to Ruby-like syntax. Am I wrong?


Who exactly claims Erlang is dead? The referenced article is SEO-infected garbage that also claims Haskell and Perl are dead.


Erlang and Haskell were never really the mainstream so not much changed for them, but Perl was "the language" at one point in time. It was everywhere, and now it's basically in the same box as Cobol - people who use it are 99% maintaining some old script. Very few young people choose to learn it nowadays, and the development got super slow due to the lack of developers. Perl 7 sounded great on paper, but I doubt it will get released anytime soon. So if not dead yet, Perl is definitely dying a slow death...


use Perl;


Well, it's 21 century, nothing dies anymore but people. But you have to admit that it's not booming the way it deserves. I wrote in Erlang a decade ago and I thought that in ten years, surely, it will eat the world. Erlang got too many things right.

However, as people from other domains started employing its principles, Erlang's value diminished. E. g., if we now write supervisors in Python that run servers, event handlers and FSMs written in C++, and the system works fine, we don't really need Erlang.


Then you need 4 languages and each of them has slightly broken interactions. Hell is the code we write.


It's only a word but the very fact the article refers to Erlang as a "coding language" and not a "programming language" is enough to give it a cheap, amateurish feel from the start IMHO.

I'd just ignore it as low quality click bait if I were you.


It may indicate that it was written by someone who is relatively young. The teenagers I know (including my own) all say "coding", never say "programming". I suspect the latter sounds "old" to them. Since the young tend to be amateurs, I whole heartedly agree that it sounds amateurish. It probably is, and we're probably getting old.


"coding", sure -- I say that all the time too. But "coding language" is something I have never heard.


That sounded odd to me too, but not being a native speaker, I was asking myself "is that how they call a programming language nowadays if they want to sound cool?". Guess not...


Erlang is designed to accommodate principles for building reliable systems in the presence of software errors, and in such capacity it is definitely a worthy instrument. It is one of the few concurrency oriented programming languages in the world.

But as Joe Armstrong himself pointed out in his dissertation: "indeed concurrent programs can be written in languages which are not themselves concurrent". You can write concurrent programs in Python if you want. Or even shell. As soon as you can spawn a process and send it a message - you're good.

And this explains why Erlang hasn't conquered the world. As soon as it has established the principles, as soon as it proved them feasible in practice, it has made itself redundant. The lessons we learned from Erlang were larger than Erlang itself.


> As soon as it has established the principles, as soon as it proved them feasible in practice, it has made itself redundant.

Features such as the memory model and lightweight processes are intrinsic to the BEAM VM and aspects of the language (such as immutability). Erlang is a mixture of no memory management, fail-fast and embarrassingly parallel without GC pauses, and getting those characteristics in another language can’t be had without fundamental and breaking changes, to the point of becoming Erlang, which seems awfully redundant.


I strongly disagree that we learned much of anything.

For example: hot upgrading. The fact that an Erlang process can tail call itself means that you can swap in a new process with the same signature at the tail call of the old process. Nobody else does this that I know of.

I can go on further. The whole set of behaviors encoded by the OTP libraries is also something that nobody else seems to do. Erlang's bit syntax is possibly the best out there even today.

I guess pattern matching finally made it to other languages. I would think that's probably about it from Erlang (pattern matching isn't exclusive to Erlang but it was one of the earlier languages with it).

None of the other programming languages seem to have learned much of anything from Erlang--and that's kind of sad.


> For example: hot upgrading. The fact that an Erlang process can tail call itself means that you can swap in a new process with the same signature at the tail call of the old process. Nobody else does this that I know of

common lisp is in a league of its own when it comes to hot reloading and debugging


Question is if Common Lisp got the idea from Erlang, or vice-versa. Common Lisp first appeared in 1984 while Erlang appeared in 1986. If so, the original point of "None of the other programming languages seem to have learned much of anything from Erlang" might remain accurate.


In the 70s Ericsson programmed their telephone switches in a proprietary language called PLEX. It had hot code swapping, so when Joe Armstrong started working on Erlang to replace PLEX in the 80s this was a requirement. Dropping a few thousands of calls just to do an update simply wasn't an option.


On the other hand, even the first versions of lisps (as far as I can gather at least) had `eval`, meaning a running program could accept external output and update itself. And this was in the 60s.

The question remains :)


Lisp would also be able to compile code to assembler, run an assembler and load the generated code into the runtime.

I would think (without knowing too much about Erlang's mechanisms) that the mechanisms of Erlang are quite different from what a Lisp runtime typically does. The Lisp runtime is just one process. Erlang is concerned with multiple processes, which are strongly isolated.


Hot upgrading is not a good example. There are infinity ways to do accomplish the same in other ecosystems, a very common example: Upgrade nginx live and without dropping any connections.

Anyhow, instead of bolting that on to your project reinventing those patterns for the millionth time you can just adhere to ̶the c̶o̶n̶j̶o̶i̶n̶e̶d̶ ̶t̶r̶i̶a̶n̶g̶l̶e̶s̶ ̶o̶f̶ ̶s̶u̶c̶c̶e̶s̶s̶ the tenants of 12 factor apps: https://12factor.net/disposability

> None of the other programming languages seem to have learned much of anything from Erlang--and that's kind of sad.

Learn you some very basic devops and you can collect on those erlang-y benefits using whatever languages/tooling/paradigms you want. A lot of it is flat out built into and managed by ye cloud providers.

Infra people squint at something like Beam/OTP and see a "control plane".


> Upgrade nginx live and without dropping any connections.

You can do that, but you won't get the new behavior on the existing connections, only for new connections. Some http servers will also curtail keep-alive requests while draining the old server processes, so you still get to pay the cost of reestablishing long lived connections, if your use case includes those.


Erlang was originally either implemented in Prolog or was even a library / extension for Prolog, and its pattern matching functionality is just "because it was all Prolog and the developers were using Prolog", so it isn't really fair to give Erlang any credit at all for that as it wasn't even independently designed.


> For example: hot upgrading. The fact that an Erlang process can tail call itself means that you can swap in a new process with the same signature at the tail call of the old process. Nobody else does this that I know of.

You can do hot loading in C with dlopen and friends, but you've got to be a lot more intentional, and have the right shape of program. I did it once for a networking proxy type thing where Erlang was inconvenient (I'd need to make NIFs to get at the data and didn't want to deal with that), but I had state I didn't want to lose or serialize while making changes to the code. Of course, I then proceded to make only one or two changes. ;)

Edit: or you can do it in C the more exciting way, by updating your mmaped libraries on disk. That'd take a lot of intentional effort to not just crash, which is what tends to happen when you accidentally update instead of replace mmaped libraries. (There's a reason to use install instead of cp)


> swap in a new process

Very technologically cool, but I'd much rather inject the new functionality into the source code. From there it can it make its way through the unit tests, integration tests, deployment, health/readiness checks, and if all that passes then I'll allow it to be called.


OTP allows multiple versions in concurrent upgrades across a cluster.

Additionally: testing, performance profiling, and debugging are natively supported.

Recommend reading:

"Designing Elixir Systems with OTP: Write Highly Scalable, Self-Healing Software with Layers" ( James Edward Gray, II, Bruce A. Tate)


Fargate allows multiple versions in concurrent upgrades across a cluster.

Additionally: testing, performance profiling, and debugging are natively supported.

Recommend reading:

"Amazon CloudWatch Container Insights for Amazon EKS Fargate using AWS Distro for OpenTelemetry" (https://aws.amazon.com/blogs/containers/introducing-amazon-c...)


One solution does not include metered connections, or Mr. Bezos compliance.

Cheers =)


Sure but that is just one example. There are lots of ways to roll your own similar setup on your own hardware. If you don't like Fargate you can go the k8s route, avoiding lock in is the whole point of it.


> Very technologically cool, but I'd much rather inject the new functionality into the source code.

Swapping in a new process and injecting the new functionality into the source code are not mutually exclusive. You can do both!

Erlang programmers are not idiots who would swap in new processes live without writing the source code for it and testing it. They write the code, the unit tests, the integration tests, deployment, health/readiness checks, and if all that passes then the deployment tools take care of swapping in the new process with zero downtime.


> Very technologically cool, but I'd much rather inject the new functionality into the source code. From there it can it make its way through the unit tests, integration tests, deployment, health/readiness checks, and if all that passes then I'll allow it to be called.

Generally, you do make the changes in source code, compile it, do all your pre-flight checks, and then hotload it. Honestly, sometimes the pre-flight check is just 'does it compile', but that's philisophical; you can put whatever process in place you want. But from my experience, it's much nicer to upgrade in place than start new, move traffic, drain old (not always in exactly that order). Sometimes, it's not realistic to avoid starting new and moving traffic, but it's nice when you can.

Most people aren't patching the VM bytecode by hand (to my knowledge).


There was that.. interview with a witch. I was a little enamoured by her.

https://aphyr.com/posts/341-hexing-the-technical-interview


> And this explains why Erlang hasn't conquered the world.

I suspect the reasons are less technical than marketing, timing, etc. It takes some experience to appreciate what it does - it's not an obvious sell like "get a webapp going in 30 min" can be for Ruby. Although Elixir/Phoenix are doing some amazing things to fix that :)

OS processes are a fine way to isolate concurrent processes if they're quite granular. It works less well when there's 10000 of them (e.g. websocket connections), and they want to share a DB connection pool, metrics, etc.


> As soon as it has established the principles, as soon as it proved them feasible in practice, it has made itself redundant. The lessons we learned from Erlang were larger than Erlang

I made a blog post around this very idea, that new languages never get the adoption that they need, because existing languages simply incorporate the best ideas from the new languages.

At which point, why will anyone change the the new language if the features that are important have been fitted into their current languages.


Those languages tend to die relatively quickly (zoomed out on a timescale, obviously), languages like CoffeeScript. When it first appeared, it was a god-send, but the good ideas were integrated into the core language, and itself went on to "die".

But then you have languages that have features that you cannot replicate to other languages. You might be able to add arrow functions from CoffeeScript to JavaScript, but you won't be able to tack on homoiconicity from lisps (and macros) to JavaScript, because it simply won't match with how JavaScript is written.

I guess my own conclusion is that languages that don't really invent much can be useful in the short term (like TypeScript), they go on to die when best ideas are integrated, but languages that are really inventive (like lisp-derivatives) end up on a different branch where ideas simply cannot be borrowed to other languages, so they stay around for the long haul.


> But then you have languages that have features that you cannot replicate to other languages. You might be able to add arrow functions from CoffeeScript to JavaScript, but you won't be able to tack on homoiconicity from lisps (and macros) to JavaScript, because it simply won't match with how JavaScript is written.

Yeah, but I specified "best features" and "important features", not "all features".

Maybe homoiconicity simply isn't that important to developers?

For example, I can guarantee you that, right now in todays development climate, there is no team anywhere that is going to allow code which has Lisp-like self-modifiying ASTs to pass code review[1].

So why would Javascript, C#, Java, etc developers bother to a) request those features, and b) implement those requests? They aren't going to use the feature anyway, no matter how powerful.

[1] Except those places who already use Lisp, because they presumably already understand it, and those teams are tiny.


But with none of the elegance of Erlang/Beam.


Erlang/Elixir + OTP is unique, and for those that don't understand the use-cases its a mystery why.

The article is an obvious troll, as popularity is not an indication of utility. =)


Well, I can't speak for all finance firms, but ours uses Erlang. Admittedly it's not used a lot, and Java and C++ are more widely used these days.


Do you think Rust will replace these over time? What are the blockers for this? Is Rust perceived as a "better" language by those in your profession but of cours e you can't rewrite everything every 5-10 years?

Would finance apps be written in Rust today if all of the C++/Java applications weren't already written?


I’m a big fan of Rust, and am currently working on a major Rust project, but I don’t think it’s currently a viable replacement for most Java apps, nor is it really intended to be.

If anything Go is closer to fitting that bill, if only because of its traditional garbage collection. Applications written in Java by definition aren’t too concerned with memory management, and Rust is objectively a step backwards for them.

Rust is a more credible replacement for C++, but there I think it will be fighting inertia. Many C++ devs are actively against changing languages. I’ve also seen people “writing C++ in Rust”, using unsafe everywhere so they don’t have to figure out how to design their code to fit Rust’s model. You then end up with a kind of worst of both worlds.


Although, "writing C++ in Rust with unsafe everywhere" does grant you the right to append "X written in Rust" when you finally do your Show HN, leading to a 26.7% increase in upvotes, so there is that.


I don't think Rust is really suited to replace the million LOC java apps that are found everywhere in finance. (The C++ ones probably ought to be rewritten in Java also, IMO. Or C#.)

The reason that such enormous software exists at banks is because the business rules are immensely complicated. All this code basically exists to deal with the complexity of the financial world, not the complexity of interacting with computers in a high performance way with correct semantics. In most cases performance is very secondary to being able to onboard programmers quickly to write oodles of code that can be modified rapidly to address changes in regulation or markets.

Java is good for this because it's a memory safe language, quite performant, super easy to learn (and quite common already) but also because you can abstract everything in a way that makes it possible to say, change the definition of currency exchange rates to account for new business rules without having to rewrite all the existing code.

Rust would require passing Box<dyn T> absolutely everywhere, at which points you are basically writing Java with extra steps. And I say this as someone who loves Rust and work primarily in Scala.


would Golang have a role in finance ? pros and cons of java vs golang in finance besides java being widespread ?


Yeah I mean in so far that Go is basically Java-like (GC-ed, statically typed, has interfaces, simple language) it could be used there too. Go targets very much the same kind of problem as Java. C# too.

I only have a passing knowledge of Go so I can't comment on it's advantages or disadvantages much, although one of the great strengths of Go (tooling) is also very much a strength of Java (at least in large companies where there will be dev env teams that set up the Java environment for you.)


Having seen actual sprawling Java codebases at banks and elsewhere, they tend to end up sprawling as they deal with (a) some specific old computer, or (b) the complexity of some dumb framework in the language itself. The business logic only approaches that level when you’re dealing with specific products are treated for things like taxes and customs, or, medical billing codes.


Banks are not the entirety of finance, but having worked on some finance backends I agree with the comment you're responding to. Finance code is legacy by default because the policies it's modeling are political and regulatory, and subject to quick changes and reversals.

Often the law as written is ambiguous and your implementation is based on interpretation of a specific regulatory body, or even a single auditor, which frequently changes. In a country as heterogenous as the US, you don't have "business logic" per se, but just a bundle of jurisdictional exceptions, that all change out of sync with each other an incompatible ways. You often need to keep multiple implementations of the same ruleset active in your system for different products, based on when they were initiated as well. Any "changes" are actually only additions of complexity, you almost never get to remove anything.

None of these things are strictly unique to finance but they are pervasive concerns there. Codebases sprawl for other reasons too, certainly. But they necessarily sprawl for this one in finance.


I mean, on some level, yes, but if all of the complexity people wrote with Java was merely collections of versioned strategy-patterns with average-level organization, it wouldn’t be sufficient for the language to have the AbstractEnterpriseBeanProviderFactoryFactory reputation that it does today.


I cannot name a single high-quality scientific computing and numerical algorithms Rust library that is not a thin layer over established C and C++ libraries.


I personally like Rust a lot, but it isn't on our radar at all. Nobody has even mentioned it.

Java and C++ are fine for what we do. What advantages (other than memory safety, which Java has too) do you see in Rust?


I know Rust is a systems programming language, but I've been playing with it for some time writing high-level stuff like REST/GraphQL APIs, websockets and web backends in general. And I have to admit it is a breath of fresh air.

No crazy enforced OOP with a gazillion classes and inheritance, no non-sense design patterns just because the language enforces you to use OOP for everything, an amazing type system, no null pointer exceptions, great compiler error messages and so on...

Of course there are problems, like any other language, but it is crazy how it as a systems programming language is so good at expressing domain logic.

Java has a much bigger ecosystem, but judging JUST by the language design, I'd pick Rust over Java without thinking twice.


Not OP and not choosing a side in this thread on the RIIR topic, but I would like to address why I like Rust over those other language - in addition to memory safety, I really like the stronger types (specifically sum types), the move semantics, and the data-race safety.


I haven’t heard any major stock and derivatives exchange providing connectivity SDK or documentation in Rust but there is always plenty of them in C/C++.


This is true. Though I see some market data providers offering Go ones.


I've worked for two fintech companies (Divvy and Mastercard) that use Elixir. It is excellent language to solve finance-related problems due to its ability to supervise code and recover from unexpected issues.


The real eureka moment for me getting why erlang and elixir are game changing was realizing how easy it is to build distributed applications using them, especially in Elixir the facility of this is just incredible. The OTP (Open Telecom Platform) as it's called is probably its coolest feature in my eyes.

https://elixir-lang.org/getting-started/mix-otp/introduction...


I heard on the grapevine once that Goldman Sachs have a proprietary distributed database built in Erlang.

Can neither substantiate nor deny but did make me wonder about all the hidden proprietary tech out in the world...


> heard on the grapevine once that Goldman Sachs have a proprietary distributed database built in Erlang

From the article, third paragraph:

Goldman Sachs built its messaging system using RabbitMQ, which runs on Erlang, in 2018. Jonathan Skrzypek (who now works for Coinbase) was running the messaging engineering team at Goldman at the time. The system had to be fault-free because the messages it was transmitting could "worth a couple million dollars," said Skrzypek. RabbitMQ was chosen because of its resilience and "reliable delivery, guaranteed delivery."


A comment to the posted-article, from "Frellus", who identifies as ex-Goldman, suggests RabbitMQ might not have been used so extensively:

Ex-Goldman person here. To the best of my knowledge, RabbitMQ was used in spots but not where reliability was critical. Instead the main message brokers were IBM MQ and TIBCO. RabbitMQ would certainly lose messages in the event of a crash, and I'm not sure it would have been used for the main message bus given the rate of events. Additionally, Erlang was used for SecDB queries quite heavily. This was the bread-and-butter of the firm (https://github.com/saleyn/secdb)


SecDB fits the bill a little better for what I had heard of. Thanks


Using RabbitMQ != Using Erlang

Also orgs like GS are huge so who knows how many corners of tech they have internally


Most of the hidden proprietary tech at Goldman Sachs was built in the early 90s, so a bit less sexy than Erlang. Impressive in its own right though.


Nubank aqui-hired the people who make Elixir? So it's not quite forgotten.


For completeness, TD Securities is/was using Erlang for an in-house GED trading platform.


To save others the Googling: GED == Global Equity Derivatives.


I wonder if rust will end up as a coding language finance forgets.


What does finance have to do with distributed systems? Finance is much more about mathematical modelling, which is an utter pain in a dynamically typed language with nothing but library support for state management.


Finance companies were doing distributed systems even before Internet existed. Yes, finance has a lot of numerical work but it has a lot more data pipeline work. The reason for this is to move market data from one geographical location to another. To make profitable investments, finance companies need accurate market data from all over the world available and processed at its datacenters as quickly as possible. For this reason in the early 1980s finance companies were laying down their own cables and writing their own link layer protocols to move data from one location to another.

Now in the 21st century this has only got more intense. There is much more competition in this space. So finance companies that can make distributed systems right and reliable stand a good chance to the most money from the markets.


Finance is a huge field. I'd say the need for mathematical modelling, in one form or another, is pretty universal no matter what you do in Finance. But there are also tons of other stuff you might need on top of that depending on what specific field of finance you work in. Distributed systems can provide the data that your model relies on, or even the platform where the model itself needs to run on.


Downtime is death, and performance is important. Distributed systems usually can help you avoid some of the types of downtime, and can help you manage throughput, although straight up latency tends to be become slightly worse.


We didn't forget it. We burned it with fire, and then buried it. Most of us still of nightmares of it. Some didn't survive. :(


Care to share one of the nightmare


I've written some Erlang and found the syntax awkward. One example of this is that variables can only be bound once. This means you end up writing:

  Val = SomeFunc()
  Val1 = Func2(Val)
  Val2 = Func3(Val1)
Yes I know you can use the functional style to minimise having to do this but you still end up seeing this pattern in real code.

Kubernetes has done a good job of replacing OTP.

That said I think Erlang is still an interesting language to learn just because of how different it is.


> Kubernetes has done a good job of replacing OTP.

I feel like this statement alone demonstrates a wild misunderstanding of what OTP does and is. Kubernetes is at best a replacement for VM-level scheduling, but without a slew of additional functionality and application-level interaction with the kubernetes runtime, individual servers are basically running blind in a container, totally unable to do anything that OTP provides to gen_servers for free.


I personally don't find this binding once behaviour problematic in any way, but Elixir specifically addresses this (it creates another variable with the same name, though)


I wish they didn't tbh, it makes me sad.

Not allowing rebinding does provide a little bit of encouragement towards factoring things better, but I have seen manys a `X1` `X2` in my time :(


its a bit of a smell, but good elixir code doesn't have to worry about rebinding for bindings in the same scope.

Jose Valim goes on as to why he made that design decision from the following: https://groups.google.com/g/elixir-lang-talk/c/w83lKZs4YS8/m...

    I will post the same answer from before: I like the explicitness of pattern 
    matching. In Elixir, as soon as I see ^foo, I know I am matching. If there is 
    no ^, I know the previous value regardless if it there is one or not, will be 
    discarded. If pattern matching is not explicit, I always need to know if a 
    variable was previously defined or not to know what is going to happen. To me 
    this behaviour is non negotiable. In my experience, it is more likely to run 
    into accidental matches than into accidental rebindings.

    Another possible limitation to the suggestion above can be related to macros. 
    Let's suppose you have a macro that stores a value in a hygienic variable:

    defmacro do_something(a, b) do
      quote do
        var!(hello, Hygienic) = calculate_something(unquote(a), unquote(b))
      end
    end

    Someone may call this macro multiple times but we always care about the last 
    value 
    of hello. If we make a distinction in between being assigned once and then 
    multiple times, the macro simply won't work. Or the macro developer would need to 
    work around this behaviour by inspecting the environment or we would need to 
    provide some sort of functionality that does it for you. Maybe this behaviour 
    could be built-in in var! or maybe we'd need to introduce something like:

    defmacro do_something(a, b) do
      quote do
        set!(hello, Hygienic) = calculate_something(unquote(a), unquote(b))
      end
    end

    We can see how this is getting complex:

    1. Use = to define variables
    2. Use ^ for pattern matching
    3. Use := for rebinding
    4. Use set! in macros when you don't care if a variable was previously defined or not

    It is also interesting to point out that the := operator won't work for rebinding 
    inside clauses. For example, how would you make the rebinding below explicit?

    x = true
    case false do
      x -> :ok
    end

    Of course, there are interesting consequences for making a distinction in between 
    defining and rebinding a variable by introducing something like the := operator. 
    We could have better control of the scope to provide better warnings or even make 
    it easier to implement imperative for loops:

    x = 0

    for i <- 1..5 do
      x := x + i
    end

    x #=> 15

    But the fact Elixir have different ways for variables to be introduced in the 
    scope, adding more rules can make the overall system very complex.


Sounds like a good old foldl may or may not have brightened up your day (with the "functional style" you mentioned) e.g.

  Pipeline = fun(Functions, Initial) ->
    lists:foldl(
      fun(Function, Acc) ->
        Function(Acc)
      end,
      Initial,
      Functions
    ),
  Initial = SomeFunc(),
  Pipeline = [Func0, Func1, Func2, Func3],
  Result = Pipeline(Pipeline, Initial).
At a certain point I grew to not mind the punctuations `; , .`

I often miss language itself given how verbose things are in elixir by comparison.

The language ecosystem is pretty good with LFE for the Lispers https://lfe.io/ and https://gleam.run/ for the OCaml inclinded.


seems more complicated than the following elixir code.

    Val
    |> SomeFunc()
    |> Func2()
    |> Func3()


I agree, or a nice `with` clause.

When strict simplicity is not the ultimately goal dare I say leaning into the monads is the way...


Why do you need the extra variables when you can pass function expressions into other function calls?


It just turns out you can't always do that in a real codebase. For example see here:

  https://github.com/apache/couchdb/blob/23efd8e5b1aa96ef01640fec03a5fedc945ba8b9/src/couch_mrview/src/couch_mrview_http.erl#L228


I really love Elixir (and somewhat Erlang) and all the mental models and how developing in the ecosystem feels like, especially with phoenix/liveview.

Having said that, nowadays anything that can't really run on AWS Lambda is a nonstarter in any company tech discussions, no matter how cool/productive it is. The operational benefits of serverless are simply too great especially when trying to scale up.


...what?

The 99.999% of tech that does not, in fact, run (or would want to run) on friggin' AWS lambdas would like to have a quiet word with you.


I thought around 80% of big tech companies are still maintaining giant Java monoliths that are attached to a giant Oracle database that nobody understands. Because it was all written 20 years ago, and there's no money in completely changing the architecture.

I've never been able to use serverless at work. I've always wanted to, but it's too risky for most companies. In my experience, they prefer to stick to tried and tested methods that have an easily predictable budget.


> I thought around 80% of big tech companies are still maintaining giant Java monoliths that are attached to a giant Oracle database that nobody understands.

I may or may not know lots of people who even develop new systems that way. In part due to them not knowing better, in part due to it being an okay fit for the problem that they want to solve and in part because they haven't and won't be fired for picking what has historically been a "safe" option.

More people have probably gotten burnt in the process of developing microservices and failing at it, than have been by their monoliths rotting over time and failing to scale, unless it's in an org with a strong ops culture and the support to do microservices right (with all of the tooling, like service meshes, tracing, orchestration and so on).

I reckon that many out there have done nor microservices, nor serverless, nor have any interest to. They're just the silent majority: https://vadimkravcenko.com/shorts/the-silent-majority/


This has it backwards - lambda is only suitable for the cloud equivalent of shell scripts, anything more complex and it becomes an unmaintainable rats nest that cannot be debugged.


Just hook up Amazon™®© X™®©-Ray™®© into Amazon™®© Redshift™®© then use Amazon™®© Athena™®©! Simple!


We should not have to live like this.


No one has to, there are 1000s of alternatives but some people chose to. Don't ask me why.


I, in fact, use Elixir serverless: I just send functions to my BEAM and it runs them for me.


I know (think) you're half joking, but I wonder if you could abuse the hot-reload system to injected your function (assuming you don't want to eval strings) into some "lambda" module and run it.


You are vastly overestimating the importance of AWS Lambda, even in web tech circles.


I don’t quite understand this.

Doesn’t the BEAM runtime also have operational benefits over “lambdas”?

Also not every application or component thereof wants stateless servers? What about live collaboration, game servers or just generally holding stuff in memory for perf reasons.


Yeah, one could easily argue the opposite: The excellent scaling characteristics of the BEAM means that there’s no need to use extreme performance compromises like AWS Lambda. You can scale an Erlang or Elixir application extremely far by just giving it more compute resources.


Well I remember this announcement [0] and there was this [1] but I'm not sure that went anywhere.

[0] https://aws.amazon.com/fr/blogs/aws/new-for-aws-lambda-use-a...

[1] https://github.com/aws-samples/aws-lambda-elixir-runtime


I've run plenty of things in AWS Lamba using Erlang FWIW.

https://github.com/alertlogic/erllambda

It is interesting in as much as the lambda functions stay hot for a period of time.


Stop it Jeff...


Time to go outside and touch grass.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: