Here's my observation: People buy products with features, not products that let you build features. Programmers are no different. Most of them will choose a language that does x over a language that can easily be extended to do x, even if the library that provides x has already been written. The fact that their favorite language doesn't do y and z doesn't matter because they don't know about y and z, and the fact that their favorite language prevents them from inventing alpha and beta doesn't matter because they think in terms of using language features, not extending the language. They want something they can use without thinking very much about it. They don't want to change it.
Of course it is hard to do programming without doing any metaprogramming at all, so new languages, like C#, have some basic metaprogramming features. But you have to market them as features of an otherwise fixed programming language or you'll scare people off. It's all about minimizing the number of things your customer has to think about. It's even worse if management is driving the choice/design of the language.
I'm not arguing things like LINQ shouldn't come with the language. I'm arguing that I should be able to add them if I want to. In Smalltalk (or Lisp, or Scheme), I can. In C#, I have to wait for Microsoft to get around to it. Look at the situation with lamdbas in Java, if you want another example.
That doesn't change the fact that I'm very glad most Smalltalks ship with ROE or GLORP, or that C# ships with LINQ. It just says that I want to be able to add features as easily as the language designer can.
I've expressed exactly this point to many people involved in C#. Partly, they do not appreciate that most PL features can be implemented as libraries in more powerful languages. Another part is they understand that blub programmers are interested in features, not libraries. Being popular means appealing to the masses.
If a language does not have closures nor more complex control flow features, you really can't implement PL features as equivalently easy-to-use libraries. IMHO, that's why most mainstream languages continue to add superficial features rather than fix the core problems in their PL.
I'm working with matlab currently. It is simultaneously the best engineering tool ever and the worstest, lousiest development environment ever. It has tremendous shortcuts for doing things quickly and powerfully but everything has been added in haphazard, half-cocked, half-assed fashion.
I can see how the two face of matlab have evolved together in the way that Perl has amazing power and horrible maintainability. I don't see that they would have to have evolved together. Ruby, for example, does what Perl does but with more sanity.
Matlab is a horrible language, although at least it has lambdas now. Have you looked at SciPy? It can't replace some of the more specialized toolboxes of course.
Matlab is a requirement for what I'm currently doing.
I'll research lambdas.
But I meant it also in saying that Matlab is the greatest engineering tool ever. You can do amazing things in 200 lines. Then you do 400 lines and get caught in some BS for days. But if you remember to do only 200 lines (even just 2 lines), then you can be a very, very good and feel very productive.
In my experience, Perl has inherent problems with readability on a line-by-line basis. I don't think that there's anything that can be added to change this - testing would prove a line does certain things but given that a test isn't complete specification, testing couldn't by itself change the readability problem.
Again, Perl would have more justification if Ruby wasn't visible
If someone references a book with author/title, PLEASE at least check a review on the web before commenting.
I was hoping that any answer might be insightful.
(Readability is mostly a function of familiarity. Everything has advantages/disadvantages, if I'd list the problems with Perl, readability wouldn't be there.)
This is mostly true. Why do people use Rails instead of Catalyst? Because they can see everything at once, and they know they won't realize halfway through their project that they are doing something wrong. People hate learning, and making sure there's nothing to learn will make a language (framework/environment/whatever) popular. Sad but true.
(I wrote a whole book on how to best use Catalyst. A year later, it's nearly all wrong. This seems to upset people, as now they "are wrong". But it doesn't really matter; they can incrementally move in the direction of what we consider "not wrong" today, and, wrong or not, it's not like their app doesn't work. Personally, I am happy that everything becomes outdated so quickly, as it means that I can be more productive, and my apps can be easier to write and maintain. But people are weird...)
I think programmers will also trade actual flexibility for an IDE that appears to make their language flexible. Are there any really popular languages today where bloated IDEs aren't "required"? (I consider Java and C# to be the "really popular" languages today. I don't use them, but millions of other people do.)
The main problem with all of this is that programmers aren't really responsible for deciding which programming technology will be used. A programmer wants macros. A hiring manager wants the ability to throw a dart at a list of names and find a programmer that knows that language. Guess whose input wins? (Combine this with the fact that people think programming is a "cool" profession, and you end up with 90% of the practitioners not actually knowing how to program or what programming is. Does it surprise anyone that cool GUIs beat complex computer science topics?)
The good news is, you don't have to be part of the problem. There is hardly any reason to use the "popular" languages. They don't have better platform support, they don't have better libraries, and they don't have better tools. So don't worry about them, they are for other people :)
On the contrary, there are lots of reasons to use the popular languages. The reasons mostly have to do with straightforwardness, lots of well-tested libraries (even if they do things in a way that's not the most clever), lots of people who've already had any problem you're likely to encounter, as long as you're doing things in the obvious way, etc.
These things together mean that using a popular language in the way that most people use it is easy -- it's the path of least resistance in a lot of instances. Using languages that are cool, have neat features (or the ability to build them), and/or have higher abstractions is fun for play, but it turns out to feel like you're swimming upstream when you have to do the boring requirements-laden task that you've been handed.
There's more than one reason for that, but collectively they add up to why I use PHP for things I just want to get working quickly, or for job-like things, and lisps or other small-community languages when I'm doing something just for fun.
> The reasons mostly have to do with ... lots of well-tested libraries ... lots of people who've already had any problem you're likely to encounter, as long as you're doing things in the obvious way, etc.
Economics has given us the wonderfully pretentious sounding term "positive network externalities" to describe products that become more valuable, the more people who use them. Simple concept, big, impressive term.
Many languages have this kind of community IMO. After a certain size, it doesn't matter really how big the community is because there is enough to sustain help. After a certain point, it only helps in finding specialized sub-communities which need to be that minimum size too. I think almost any language that a person here would consider for some project or another has that community, and you'll really notice the difference w/ languages that don't. For example, SML is a language that is so tiny that you can really notice the lack of community (and free documentation...) compared to, lets say, python, or objective-c.
>The good news is, you don't have to be part of the problem. There is hardly any reason to use the "popular" languages. They don't have better platform support, they don't have better libraries, and they don't have better tools. So don't worry about them, they are for other people :)
This is why I'm not interested in Clojure, regardless of how nice a lisp it may be. Java libraries are poorly-designed, the prospect of wrestling with them again brings back memories of frustration. And you can bet that no one will ever write better ones, because the community will discourage people from "duplicating functionality."
"Java libraries are poorly-designed, the prospect of wrestling with them again brings back memories of frustration."
One of the best things about Clojure is how it makes Java libraries suck much less. For example, line-seq turns a Java BufferedReader into a lazy sequence of String's, one for each line in the file.
You still have the suckiness of the multiple lines it takes to instantiate and clean up the BufferedReader, but macros like with-open makes that suck a little bit less, too.
Stir in some macro-fu of your own, and now you have the benefits of Java libraries minus the boilerplate suckiness. Basically, it turns Java into a "your language features are my libraries" language, no mean feat.
So your criticism of Clojure is based upon unfounded assumptions about what the community will do in future? Maybe you should look at what libraries are being created before you leap to conclusions?
There are many Clojure libraries that attempt to reimplement existing Java libraries. For instance, Ring is a HTTP specification for Clojure that reimplements much of the functionality of Java servlets.
"Most of them will choose a language that does x over a language that can easily be extended to do x, even if the library that provides x has already been written."
I nominate Ruby as the first "your language features are my libraries" language to become legitimately popular. This is likely due to the fact that Rails provides the libraries that most web programmers want, in addition to the cool language features. But it also seems that the "Rails community" is enamored with how easy Ruby's flexible syntax makes it to extend the language in interesting ways. Thus, all of the fuss over adding methods to base classes, and whether or not, or when, it is a good idea to do so.
So I would say that Ruby has brought the idea of language features as libraries into the mainstream, as much as anything as geeky as programming languages can be considered mainstream.
If a language includes more than just processing directives and code organization then that becomes more of a language/platform/framework than just a language. Often these are geared to a specific domain such as desktop or web software and try to give you tools that you'll most often need.
If you don't need additional libraries then you are probably doing things that the authors of the language/platform/framework thought would be necessary.
The other language language/platform/framework, which I am familiar with, is ColdFusion. If you are doing web development and fit with-in only the functionality available to you by default then you are fine. If you want to abstract away certain things or do more than average web development then outside libraries are necessary, especially in the user-interface side of things.
Well, some languages have so much apparent power that it's easy to be seduced into thinking you hardly need libraries, since you can just build it yourself right quick. Common Lisp, for example.
This is another thing that is nice about languages with simple syntax (like Lisp and Smalltalk) -- language features and libraries can "look" exactly the same. I think this is one of the big mental barriers to entry for users of other languages. If you use Java or something, adding something like a langauge feature would just be so difficult that you wouldn't even try -- you would use an XML file or something. With Lisp or Smalltalk, adding something that looks like a language feature is so easy, that you wouldn't ever think of doing something else. As a result, Lisp and Smalltalk have a lot of languge features ("loop") that really aren't.
From the standpoint of an editor hacker, I love simple languages. When I write Lisp, and create some "new construct", I just follow a few simple rules and my editor can understand the code perfectly. (Name your definition-creation macros define-something-or-other, and emacs will syntax highlight them correctly. Perfect.) When I am extending a more complicated language, the editor hackery is much more involved.
A bit of backstory, Perl now has a module called Devel::Declare, which lets you hook the parsing machinery and run your own code when you see certain things; this basically adds macros to Perl. "It's not a source filter." A number of people have taken advantage of this newfound extensibility to add some great features; try/catch syntax, class definition syntax, etc. I became a heavy user of this stuff, and I wanted emacs to understand it. 3 days of hacking and 200 lines of code later, it does. But that's a lot of effort to go to. When extending Lisp, I had to do nothing. When extending Perl, I had to hack on my editor for a few days to get everything right. (And don't ask, "what happens if you don't use the module that introduces the syntax; does emacs detect that?" because you won't like the answer. But, try typing "define-thing-that-is-not-defined", and emacs will syntax-highlight that as though it's a builtin, and not as random garbage you just entered. Oh well :)
I think this must be a mental barrier for those who are not comfortable with both the internals of their language and with the internals of their editor, and that's why users rely on The Language Designer Gods to give them features.
This is another thing that is nice about languages with simple syntax (like Lisp and Smalltalk) -- language features and libraries can "look" exactly the same. I think this is one of the big mental barriers to entry for users of other languages.
It happens in C++, too, which has an extremely complex syntax. C programmers are used to thinking of operators like + and - as language features because their meaning is defined by the language and can't be changed by programmers. C++ operating overloading continues to be extremely controversial because people who are used to C get angry that + and - can "unexpectedly" have meanings other than the meanings defined in the C language.
I know that's a special case, nothing like the general mutability of Lisp syntax, but I just wanted to point out that the differences pro (superior freedom/expressivity) and con (fear of confusion) happen whenever one language is slightly more programmable than another, not when a language has simpler syntax. (Of course, any sane person designing a programmable language would prefer minimal syntax.)
>I think this must be a mental barrier for those who are not comfortable with both the internals of their language and with the internals of their editor, and that's why users rely on The Language Designer Gods to give them features.
But programmers who have that mentality probably should wait for The Language Designer Gods to give them features. These are the same people who should not be allowed to write frameworks and should not be allowed CPP macros and the use of #doesNotUnderstand: in Smalltalk.
Maybe. I think if you give someone a tool, they will eventually learn how to use it. If you give them the possibility to make a tool, though, they might not ever figure out how to make the tool. There are two different levels of thinking involved; more people can do the former than can do the latter. Some people can do neither, though, I will give you that :)
I think one of the benefits of features, rather than libraries, is that someone can be expected to know what it does.
In C#, ?? should be understood by anyone who knows C#. In Smalltalk, ?? will be understood by the implementer and then need to be explained or learned by every fresh set of eyes. (Yes, they might expect it mirrors C#, but the problem still exists. The eyes know Smalltalk, not C#.)
In Smalltalk, if you're curious about ??, you'd be able to find it instantly, and read the method. As you know from the article, it's fairly short and would be obvious to a competent Smalltalker in seconds. If that's not enough just take a second to type in
nil ?? [ self halt ]
And right-click "Debug-it." Debugging in VisualWorks is so painless, people actually write code in comments for people to understand by debugging -- and people will even follow it while in another debugger session! (And even doing that a 2nd, 3rd time is just as easy!) To heck with comments or API docs, you can see how everything works and tinker with it!
Your division between access by the implementer vs. the "fresh set of eyes" is a misconception you're taking from C#. There is nothing in Smalltalk that demarcates the "implementer" from the regular programmer. It's "Turtles all the way down!" You can actually implement a debugger as an ordinary Smalltalk app, and in 5 minutes, you can be browsing a stack. There is little difference between programming and meta-programming.
In other words, Meta-programming is easy and natural, not esoteric.
You just see Smalltalk as a funny sort of Blub where no one knows what the language/library will look like, because you see things in terms of the programming you know: Blub. Take it from me, I've been to dozens of Smalltalk shops over a decade: this is not a problem. It's sometimes a huge advantage! One can create very powerful domain-specific languages and tools this way.
In theory, if the feature provided by the library is very important (apparently it is important enough for C# implementors to build it in the language), you can expect that the library will eventually becomes standard or quasi-standard. For any language, its programmer is expected to learn not only the language features but also standard libraries, so eventually the difference of "knowing" part will be leveled out.
In practice, there are two factors that work in opposite directions, and which factor you're looking at changes pros and cons of language feature vs. library discussion.
* Time difference - for language features you have to wait until implementors implement them. For libraries you can start immediately. That gives you time advantage, sometimes for as much as years.
* Pressure difference - to make it standard language features there are lots of pressures to make it right, and effort is taken accordingly. Similar amount of effort is needed to make a library feature into standard position. But for those who wrote that library feature in the first place, usually their need is fulfilled easily, so there may not be enough incentives for them to push the feature into standard.
I feel the time advantage the 'library' approach gives me is crucial, so I prefer flexible languages (specifically I use Scheme). But in this side I do see the effect of the latter---there are tons of libraries, each of which is "good enough" for a specific circumstance but a bit short to be truly general, reliable solution.
One of the things that distinguishes a good programmer from your average one is a detailed knowledge of the language's standard libraries, even if they're de facto standard ones. How many times have we seen C that re-implements memrchr(3) or strcspn(3), often with bugs.
Which is implemented via a static class extension method. C# doesn't ship with a Zip method, but adding one is trivial:
public static IEnumerable<TResult> Zip<TFirst, TSecond, TResult>(this IEnumerable<TFirst> first, IEnumerable<TSecond> second, Func<TFirst, TSecond, TResult> resultSelector)
using (IEnumerator<TFirst> e1 = first.GetEnumerator())
using (IEnumerator<TSecond> e2 = second.GetEnumerator())
while (e1.MoveNext() && e2.MoveNext())
yield return resultSelector(e1.Current, e2.Current);
}
That makes it a library feature of C#, not a language feature. Only the LINQ syntactic sugar is a language feature, and I would argue the decision to include such a syntax was a bad one.
Actually, still true. You're right that LINQ expressions are syntactic sugar for C# calls, but that syntactic sugar depends on lambdas (introduced in C# 3) and expression trees (also introduced in C# 3), among others. You cannot write LINQ as you know it in C# 2. That's my point.
Heh. You still have anonymous delegates in C# 2.0, and you can roll your own expression trees if you really want them. Same thing for yield and extension methods. Of course your code will be much more verbose and unnatural, and you won't have as much compiler support.
Yes my friends, we are deep in the Turing Tarpit, where the boundary between language and library is blurred, baroque architecture looms menacingly in the mist, and the bare metal is somewhere far below the fetid surface of the swamp. I hope you brought your safari hat and your elephant shotgun.
Ok, but using the previous version of the language to demonstrate your point is much less interesting. Additionally, there is the issue of clarity: when I read "C#" I tend to assume that means the language as it exists today, not some previous version.
You completely missed the point of the article, which was that I can add these features to the language without waiting on the language designer. Microsoft added three key technologies to C# 3, one of which was syntactic sugar for the other two. That's irrelevant to my point.
The central issue for me is that I don't like getting bogged down in the meta. I hate the idea that a lot of what programmers do is program around programming, as opposed to actually solving real world problems with their programs. Of course the end goal is solving said problem, but you have to do a lot of meta to get there. Its unfortunate how much of our libraries and so forth are just pure architecture.
I like language features because they provide a clear distinction between the meta (things that I have to do due to the limitations of my execution environment) vs. the actual steps I'm taking to calculate some result or provide some service. When I see a language keyword or operator its clear to me that this is glue/implementation, and when I see a function call its clear to me that this is an actual step related to the algorithm. Classes and modules and templates and all these "objects" ideas have little to nothing to do with algorithms, they are simply the way we choose to implement those algorithms.
This varies language to language of course. Smalltalk has the advantage that I can create library features that "look" like language features. This is a big plus in my book. I think if I were using Smalltalk I would probably not have a problem with this at all. However, taking an example like JavaScript, I very much dislike this style (which many js frameworks currently use):
In a cursory view, the actual work (calculating cosine), looks exactly the same as creating a class, there's no visual distinction. Additionally, this encourages every library under the sun to make their own slightly different version of the class create/subclass method which makes it difficult for IDE writers to be able to pick up on these semantic elements of your program, since it really just is a function call and nothing more.
I've been deep in the depths of hacking a new feature into an existing language - adding a reflective/referential data definition language on top of haXe. (It's called XReflect, if curious you can get it off haxelib)
Essentially, the problem I keep running into with referential data structures is that haXe, like most popular languages, will only let you set them up in an imperative fashion, one dependency at a time, which tends to both spread out and bloat application code. What I want is to write my dependencies in any order and let the resolution be sorted for me.
I'm pretty sure a language with AST access and quoting like Lisp makes it relatively easy to write something along these lines, but in haXe, I had to write about 500 dense lines of code, most of it going to parsing and fighting the limitations of the type system. It still has some major bugs that I'm trying to resolve with a refactor at the moment.
The funny thing is, I know I'm not just fighting haXe here, I'm also running into the limitations of the platforms that haXe compiles to. The newest strategy I'm applying to resolve references is to add a "__parent" field to the reference stubs, which refers to the container(an array or record type) of that reference. I use that later, after all references are resolved, to drop the final value into the place of the stub, but as it turns out this causes difficulty printing out the working structure. Neko will recognize the value of __parent as "....", but Flash 9/10 will recurse into the field again and get a stack overflow.
So I had to add more code to write a pretty-printer which is aware of such traps.
The problem with languages where everything is possible syntactically is that everything is possible syntactically. You're no longer dealing with a language that everyone speaks and understands -- instead you have to learn to local dialect just to get started. If the programmer before you was a total git, you'll be in more trouble than if you'd use a more constrained language. This is why most languages after C++ have avoided operator overloading.
> This is why most languages after C++ have avoided operator overloading.
Except for C#, PHP, Python, Ruby, Perl, Haskell, Io, Lua, Scala, Groovy, D, F#, and even Visual Basic.
Really the only major post-C++ languages to avoid operator overloading were Java, JavaScript, and Objective-C.
I've seen an argument that Java learned the wrong lessons from C++. They saw how complicated operator overloading was in C++ and avoided it, despite the fact that the worst complications in came from mixed pointer/value/reference calling conventions (which of course wouldn't affect Java). C# proves that it's completely possible for a Java-like language to have safe and easy operator overloading.
If Java had made primitives syntactically behave like immutable Objects with a proper implementation of equals(), Russell's suggestion would've made obvious sense and the language would've made more sense overall. The road not taken.
How does this overcome the need for two kinds of equality for objects? i.e. whether two pointers refer to the same address vs. whether they point to different addresses (or the same) that contain objects with the same value, according to that object's definition of equals().
Do you mean that the present == for primitives should be replaced with .equals(), and the == for object reference equality should be replaced by perhaps === or .refEquality(), and then the .equals() for objects should be left alone. and then, == be added as syntactic sugar for .equals().... Are those assumptions what you meant?
If so, the same outcome can be achieved with by changing object == to ===, and object .equals() to ==. However, there are reasons for treating primitives as objects other than this.
I expect you've a model in mind from some other language that resolves this in along different lines (C# would be one), and so you haven't felt the need to articulate the assumptions needed to make it work. If I've guessed wrongly, can you elaborate please?
The overwhelming majority of actual Java code needs at most one equality operation defined per class. Hamming compression and convention suggest that it be called == . Extensibility suggests that it desugar into .equals() or something like that.
For immutable objects like ints or Strings the implementation of equals() should compare internal values, because reference equality for passive immutable things is practically never useful. This should already obviate the majority of explicit calls to equals() in actual Java code, because most of them are comparing Strings anyway.
For mutable objects the default implementation of equals() should compare object identity (hashCode), but be overridable if the user wants something exotic.
Algorithms and data structures relying on equality, like hash tables, should provide the option of supplying the equality operator or hash function at construction time, just like printf frees the user from the need to commit to one Integer.toString() implementation in perpetuity.
This all sounds like a reasonable scheme to me. I can flesh it out in more detail if you request.
Yes, thanks, that's what I thought. BTW couple minor points: hashCode doesn't necessarily distinguish between objects (though most implementations would); and an exception for mutable objects is collections (e.g. a Set is mutable, but you want to be able to compare sets).
Yes, I'd want to compare sets, but using == for that feels to me a dubious programming practice, akin to using + for set union. A special method like hasIdenticalContents() would make me more comfortable, the longer name removing equals-ambiguity for readers and reflecting the nontrivial computation behind the scenes. And that would mesh nicely with other set operations like difference and containment that would be getting non-canonical names anyway...
Also, if you make Set.equals() depend on contents, you'll have to change the implementation of hashCode as well, wreaking havoc on sets containing mutable sets. So you really should only override equals() if your class is immutable or you're feeling especially exotic.
Your comment made no sense, then when I started to write a reply requesting clarification, it refreshed, and you'd added more. I feel much better now that it makes sense; and I'll watch out for the live-edit-comment in future (I do live-edit myself too).
But I'll say that naming is tricky, since programming languages are really human languages, and as such are affected by the familiarity of their specific users as much (or, at times, more) than pure logic. Highly illogical, I know.
EDIT: The present implementation of Set.equals() depends on contents; it's independent of hashCode, and there's no problem with sets of sets.
I think this speaks to the value of the "local dialects" that you have had a chance to learn. In Smalltalk and languages of similar capability, it's possible for a project to develop domain-specific "local dialect" that has prodigious power. Any project that's really worth its salt develops these. Most just develop their own way of doing the same inefficient stuff.
Meta-programming is much more powerful than just plain programming. It's also not as widely understood.
I think his point is when you have to use parts of other projects. Unless you are operating in a complete vacuum, this will still be an issue when using other libraries, etc. Also, the "coding practices, constraints, and code review" can be used to hand wave absolutely every language shortcoming aside.
Languages are a way to communicating both to other humans and to your compiler, and doing both well is really hard. Sometimes syntactic sugar makes it easier for humans even in powerful and dense languages: think of the "do" notation in Haskell.
And no, you can not put every language "feature" in a library: what about features like static typing, isolation of side-effects... Even features like threads can be a library, but the compiler has to know about this library and its particular semantics...
So yeah, we all know languages like Lisp or Smalltalk are very expressive. But current C# is also quite powerful and relatively pleasant to work in, and it's goal is rather to bring more features to programmers and to evolve this "blub". And of course I hope someday the average programmer will write in a Haskell-like-but-even-better language :)
[EDIT - this 1st paragraph is wrong and retracted]<wrong>
The post misses a large part of the point of LINQ to SQL, which is that it constructs optimized and type mapped queries against an external data source, rather than filtering an in-memory ORM object -- and it manages to do it with static type checking.</wrong>
Adding a feature like ?? to C# isn't hard in itself, at the parser/CodeDOM level. What makes it necessary to be careful about limiting the scope of the language is the tooling, the IDE, the ecosystem work.
I thought I spelled this out quite clearly, but the ROE query provided does exactly that: it builds an optimized SQL query and executes it in the database. In the example provided, the SQL generated will roughly be "select firstName from people where people.type = (whatever the adult enum is)". To do so, the lambdas don't actually take each row of the database; they basically take abstract syntax trees that record what messages you send to it. At execution time, the AST is optimized and sent to SQL. This can leak at certain places--for example, calling a real method on the object, rather than comparing attributes--but they're identical to the places LINQ to SQL falls apart.
Of course it is hard to do programming without doing any metaprogramming at all, so new languages, like C#, have some basic metaprogramming features. But you have to market them as features of an otherwise fixed programming language or you'll scare people off. It's all about minimizing the number of things your customer has to think about. It's even worse if management is driving the choice/design of the language.