But you aren’t the worst programmer either. You try to do your work well and you read a few technical books a year. You attend a user group or two when it fits in your schedule. There are a few technologies that you understand fairly well, and others on your team look to you for help in your areas of relative expertise.

The problem is, you are in a rut. You aren’t gaining skills as quickly as you were last year. You aren’t exactly thrilled about your current job, but not so dissatisfied that you want to take the chance of trading your known problems for unknown problems. What to do?

You need to be the worst. It’s time to get that job that you think you don’t qualify for, and work among developers that are all better than you. Don’t just apply for a different job, apply for the best job you can.

After I had some experience as a Java web developer, I somehow managed to get on as a consultant with Pillar Technology, which hires some of the best software developers in my area. Some of them have written technology books, they regularly speak at user groups, they contribute to (or create) open source projects, they host code-retreats, and they all live and breath code. It was obvious to me that I didn’t really belong with these guys.

To say that you will be pushed and challenged in such a situation is a truism, but it is amazing to me how much my view of the IT world changed from a relatively short time at Pillar. During that time I gained significant work experience a variety of technologies that I hadn’t used before, with opportunities to learn from real experts, and I experienced agile software development, TDD, pair programming, and continuous integration along with a team that was very committed to delivering excellent software. I also found myself spending my own time learning technologies that I wasn’t working in, like Haskell, Python, Ruby, RSpec, Git, and Vim, because there was constant conversation about new or interesting languages, technologies, and tools.

In the end, working at Pillar did not make me as good as the Pillar guys, in my view. But it gave me an opportunity to understand what great developers look like, and to see what sort of effort it takes to produce excellence. And their enthusiasm for excellence and delivering valuable software is infectious.

If you try to be the worst you might fall on your face. I had one interview with an excellent start-up in the area where it was clear that I was out of my league. I had suspected that this might be the case, but I made it through the phone screen, so I went for it. That isn’t a fun experience, and it comes at the expense of time and emotional investment, but you simply can’t know until you try. And really, if you aren’t concerned about the possibility of being exposed and embarrassed, then it is likely that you are aiming too low.

You’ll never be the best. But if you have the courage to be the worst, you may find yourself significantly better in the end.

This is not a book to read to learn Java, Bloch assumes throughout that you have a familiarity with the language. It is also not a book to help you take the next steps after learning core Java, such as learning popular APIs such as JDBC, Java EE, or Swing. Effective Java is unlikely to equip you to accomplish any programmatic goal in Java that you were unable to code before reading it.

If you are overly pragmatic, it may seem that this book meets no real need. But if you have learned several languages, the purpose of this book, and the importance of books like this, is obvious.

Why read this book? To learn how Java ought to be written. Any programming task can be coded in a variety of ways, and Effective Java is about writing Java code that is understandable, maintainable, and fits with the design of the language. In other words, Effective Java will teach you how to work with Java, rather than working against it.

As for when to read this book, you should get this book as soon as can write code in Java. This is the book to get when you find yourself thinking, “I know a way to do it, but I’m wondering what is the right way.”

The book is structured with 78 “items” divided into ten sections, including “Classes and Interfaces”, “Generics”, “Enums and Annotations”, “Exceptions”, “and “Concurrency.” Each item takes a narrow topic and argues for a preferred practice. Some items in the “Classes and Interfaces” section are “Favor composition over inheritance”, “Design and document for inheritance or else prohibit it”, and “Prefer interfaces to abstract classes.” Each item is supported with concise arguments and brief examples, and often suggestions of possible exceptions to the rule. This is a very easy read, due to the short chapters and Bloch’s clear exposition. Each entry is like a blog post, easily digested in a short amount of time.

This book is a classic that will continue to fill an important need for as long as Java is relevant.

The Internet is full of praise for men whose professional successes come at a terrible cost to their wives, and to their children. Meanwhile we wring our hands at the costs of fatherlessness in our nation. Let us take our hats off to Jeff for his manly choice to do the humble, necessary work that does not lead to public praise.

Here’s to you, Jeff, and to all of the sane ones.

Jeff’s work on Stack Overflow will always have special meaning to me, because of the essential role it played in my vocational change. When I was starting from zero at age 36 my only source of programmer culture was the Internet, and Stack Overflow was the most essential part of that. When I was only able to manage one hour a day for programming and I had know idea what I was doing, the ability to get correct answers to noob questions quickly would have been worth significant money. The pace of my progress would have been substantially different without SO, which was huge to a father of four that had real questions about how he would provide for his family.

So thanks for everything, Jeff. Thanks for helping me to care for my family. And thanks especially for caring for yours.

Given the surprising popularity of that post, it seemed reasonable to repackage the information in a format that is more fitting for a blog post.

find and grep are incredibly powerful. But many never learn a fraction of what they can do. Did you know that grep can print the surrounding lines? Or that you can search for files based on permissions? Here’s a collection of examples that might expand your notion of what can be done with find and grep.

Most find commands follows the pattern:
$ find [PATH] [expression]

where the expression consists of one or more of the options shown below. find is recursive by default so the command $ find . would output the entire directory tree at the current location.

Find files named Account.java in src directory:
$ find src -name Account.java

Find case insensitive in current directory — matches README, readme, ReadMe, etc.
$ find . -iname readme

Find by wildcard name — NB: quotes keep shell from expanding wildcards, it is best to always include them.
$ find /etc -name '*xml'

Some exclusions — -not can be replaced with !
$ find . -not -name '*java' -maxdepth 4

Find with or
$ find . -name '*java' -o -name '*xml'

There are several ways to do stuff with the results of your find.  I prefer to use the more intuitive xargs rather than -exec, as the -exec syntax is strange. But -exec gives you the ability to choose whether you want the second command executed once (+) or once per result (\;).
$ find . –name '*.java' | xargs wc –l | sort
$ find . –name '*.java' -exec wc –l {} \; | sort
$ find . –name '*.java' -exec wc –l {} + | sort

It is often useful to filter by type. Using find is more reliable than parsing ls.
$ find . -type f
$ find . -type d
$ find . -type l

To find all directories in current directory:
$ find . -type d -maxdepth 1

Do you want to see the files that have changed today?
$ find . -mtime -1

Or the files that have changed in the last 15 minutes?
$ find . -mmin -15

Sometimes it is convenient to find the files that are newer (or older) than a certain file.
$ find . -newer foo.txt
$ find . ! -newer foo.txt

Of the files that have been modified after a certain date … or between two dates.

$ find . -type f -newermt '2010-01-01'
$ find . -type f -newermt '2010-01-01' ! -newermt '2010-06-01'

You can also filter by size. Note that the - finds smaller files, + finds larger files.
$ find . -size -1k
$ find . -size +100M

I don’t know that I’ve every included permissions in a search, but it could be useful.
$ find . -perm 644
$ find . -perm -u=w
$ find . -perm -ug=x

A grep command typically takes the form
$ grep [options] [string/regex] [file or path]

For example:
$ grep 'new Account' *.java

Note that you do not quote a wildcard expression for grep, in this case we want the shell to expand the expression.

grep is not recursive by default, this is altered with the -r flag.
$ grep -r 'Dao[Impl|Mock]' src

I use the following flags commonly:
-i Case insensitive
-w Restricts to word matches
-n outputs line number
-c outputs count of matches

Grepping for multiple terms is useful in theory, but I rarely do it:
$ grep -e foo -e bar baz.txt

Whether searching through source code or log files, the ability to display the surrounding lines is often useful. For example:
$ grep -r -A 2 foo src

will display the lines that contain foo and the two subsequent lines. Similarly, you can use
-B before
-C centered (lines both before and after found text.)

That’s all for now, I hope that this helps you find something soon.

While Davy Crockett and others explored the American frontier, Samuel Morse lived a much more civilized life in Boston. But no one could accuse Samuel Morse of avoiding a challenge, he lived on the cutting edge of a different frontier, inventing the telegraph, Morse code, the telegraph relay, among other things.

Recognizing the different types of frontiers is necessary. It is immediately absurd to consider developing the telegraph while living off of wild game in the mountains of Tennessee. But it is not immediately obvious to a newcomer to information technology that many interesting programming jobs rely on somewhat boring, established technologies. Or that much (most?) useful software was written in languages that were no longer exciting at the time the software was created.

If you want to have a rewarding and interesting career, it is necessary to head for a frontier that is exciting to you. But you may find that in order to make the progress you want, it is necessary to pull back from some other frontiers, and enjoy the benefits of civilization.

So don’t let anyone tell you that all the cool kids use Clojure, F#, or CoffeeScript. Other people are building great things and solving hard problems Java, C#, or C++. Sure, some stick with the familiar because they don’t want to learn anything new. But many others are too focused on inventing the telegraph to leave the laboratory.

Google, the search advertising giant …

Can we stop spreading this foolishness? Yes, Google, FaceBook, Twitter, and many other web-based businesses use advertising as a primary source of revenue. But strangely, I’ve never heard anyone refer to NBC, ABC, and CBS as advertising companies. I’ve also never heard anyone call American Electric Power a billing company, or Toyota a sales/financing conglomerate. We don’t categorize companies by the most naive interpretation of their revenue model.

But it’s worse than that. Google doesn’t offer the most basic services that an advertising company would offer. Will they plan your campaign for you? Help you identify your target demographic? Produce ad copy? The primary advertising related service that Google offers is a place to display your ads. Notice that this is a service that actual advertising agencies do not offer.

If we think for a second about what other companies offer this service (TV, radio, print media), we can recognize that it would be far more accurate to call Google a media company. But don’t do that either, of course.

If you are concerned about the power and influence that Google has, as our primary distributor of internet information, say it. Just slapping an inappropriate label on the company will erode your credibility and weaken your argument.

So don’t be lazy, if you are going to read this book, set up an interpreter, and start coding.

The second thing to know about this book is that it does not attempt to teach you the Scheme programming language. Which is good news, because you don’t really want to be a Scheme programmer. What this book does teach, however, is of interest to every programmer.

This book is about recursion, and recursion is not optional. Many problems are most effectively modeled with graphs and trees, and if you aren’t able to identify and implement the natural recursion that exists, you will write code that obscures the actual problem, and is harder to write, debug, and maintain. So if you are not comfortable with recursion, this book is an immediate need.

Of course, if you are a programmer that reads blogs, you wouldn’t admit to yourself that you are uncomfortable with recursion, so let me say it another way. Working through The Little Schemer will allow you to quickly identify the basic patterns of recursion problems, and to structure your code in the way that will most clearly reveal that pattern.

I encountered recursive thought first in pure mathematics, where very similar thought is required in mathematical induction. Encountering recursion in programming was like meeting an old friend’s identical twin. The similarities were enough to make me feel more familiar than I was. As a result, my recursive solutions always felt ad hoc, and distinct to the particular problem, rather than following a well-established path.

Regardless of your background, if you are a programmer, or hope to someday be a programmer, and you have any hesitation in implementing recursive solutions, you should work through this book.

I disagree, and think that there is a place for certifications, but there is some confusion about what you can and cannot get out of them, and when they may be beneficial to your career.

What a certification cannot give you

Expertise

This should be obvious. No one becomes an expert at anything by reading books and taking tests. Expertise comes through hard work, and it likely includes reading books, but you don’t have it until your peers recognize you as an expert.

Programming ability

Don’t imagine that a technology certification will serve as some sort of boot camp / personal trainer that will whip you into shape, increasing your ability to solve programming problems. Testing for the creative thinking and problem solving ability that is essential for programmers would be extremely difficult to do, and the exam authors don’t really attempt that.

Evidence of competence

If you want some letters after your name to prove you will be competent, forget it, it isn’t possible, and you don’t want to work for a company that thinks like that.

What a certification can give you

An organized curriculum

This is one of the biggest benefits. When you are in a new environment you don’t know what questions to ask and the relative importance of the various things to learn. When you don’t know what you don’t know you aren’t in the best position to manage your education. Using the list of exam topics, perhaps along with an exam study guide, can provide great structure to your efforts.

Confidence in the rules of the game

While being a certified programmer doesn’t mean you are a good programmer, it should mean that you understand which errors will show up at run-time, and which will happen at compile time, which is valuable. There are far too many programmers out their that using guesswork and superstition when it comes to the workings of their primary programming language. A certification can give you confidence regarding the way that code should work.

An increased chance of interviews for your first or second job.

When you are trying to distinguish your resume from the resumes of others that are competing for junior-programmer jobs, a certification can help. It shows some initiative, a desire to improve yourself, and it suggests that you are familiar enough with a particular language or technology that you can be quickly taught how it is used in the real world.

Should I get a certification?

With the above in mind, it should be easy to decide if a technology certification is good for you. Are you trying to enter an unfamiliar area of technology, and struggling with the documentation? Does your resume seem devoid of technical qualifications? Are you early in your career and unsure what to learn next? If so, a certification could help.

On the other hand, if you are an experienced programmer looking to improve yourself, or hoping to qualify for a better paying job, then a certification is entirely the wrong approach. Some better ideas:

• Find your week spots in the Programmer Competency Matrix, and remedy some of them.
• Dive into the documentation of one of your favorite tools, become the expert on it.
• Read a classic programming book. (some ideas)
• Build something cool.

My experience

I took the Sun Certified Java Programmer (SCJP) exam before ever working as a programmer when my resume was a Ph.D. in mathematics and absolutely nothing technical. I’m convinced it gave me credibility as someone that wanted to become a programmer, and helped me obtain the first interview. Once on the job, it was extremely beneficial to have a well-formed understanding of the language.

About six months later I took the Sun Certified Web Component Developer (SCWCD) exam, which really helped me to understand how Java web applications, and web applications in general, work. I doubt anyone that has hired me since then has cared about that, but employment wasn’t my primary objective.

SCJP, SCWCD, MCTS, and others can be helpful in beginning a programming career. And they might hurt your chances of getting an interview at Google. But when a newbie or wannabe programmer asks you about certs, please keep the benefits they do offer in mind, and answer the question in a way that makes sense for his level of experience and short-term goals.

The bad news is that figuring out what experience to get can be paralyzing. Read the job ads and you find that experience with frameworks like Spring, Struts, and Hibernate are required for most Java jobs. Read forums and blogs and you find that the cool kids that get the startup jobs all have their projects available on GitHub or BitBucket. Some say that mobile jobs are hot. Others say that their are no decent jobs in Java or .NET, that you should be investing in Ruby, Clojure, Scala, and Node.js.

What if you don’t know where to begin? Let’s reduce the pressure by considering a list of things that you don’t need to have before getting your first programming job.

#1: Cool projects to display

If you’ve built something impressive, great, make the most of it. Most programmers haven’t, and this will distinguish you. Try to find an employer that cares about such things, as you’ll enjoy working with others that build stuff in their own time.

If you have nothing to display, don’t sweat it, most employers won’t ask. If you have an idea that you would love to try, go for it, you’ll learn a lot. If you don’t have any ideas right now, focus on learning the details of your primary language, and work on smaller exercises targeted to the concepts that you are learning.

#2: A Computer Science degree

A CS degree won’t hurt, but you’ll do just about as well with a bachelors degree in mathematics, physics, engineering, or other majors that require some rigor and logical thinking. If your degree is in English or psychology, then you have a greater burden to demonstrate that you can think logically.

If you have no college degree, then you can forget about employment at large companies until you have some years of experience, and having work that you can display publicly now becomes a necessity.

#3: Knowledge of multiple programming languages

Knowing a variety of languages and paradigms is a necessity as your career progresses. But you can get your first job with only knowledge of one language, particularly if it is Java or C#. If your one language is Python or Ruby, then I strongly recommend that you learn Java or C# immediately, as I’ve never known anyone to get a Python or Ruby job for their first job.

NB: I’m not counting HTML or SQL as “programming languages” here, you should definitely consider a working knowledge of these necessary for your first job.

#4: Experience with any framework

If you are looking for a job as a Java web developer, and you see Spring, Struts, or Hibernate listed in virtually every job ad, it isn’t crazy to think that you should learn one or more of these. But don’t bother.

First of all, these frameworks are created to solve the problems of large projects, problems that you haven’t experienced. A far better use of your time before getting a job would be to experience those problems by building a Java web application that uses Servlets and JSP, and connects to the database with JDBC. Second of all, either you have professional experience with a framework or you don’t, any knowledge you gain with tutorials and toy projects will not be relevant on your resume or in an interview.

#5: Experience with multiple platforms

These days we hear a lot about mobile development, do you need to build an app? Building a GUI desktop application seems so fundamental that it shouldn’t be overlooked, right? Don’t get distracted. The vast majority of entry-level jobs are in web development, so that should be your focus. I’ve still never built anything for a mobile device or the desktop, and it doesn’t seem to be getting in my way.

Aren’t all these things important?

Yes, they are important, and each of them can help you in your job search. My contention is simply that none of them are necessary for the first job. Obviously, some of these are essential for advancing your career.

So what is necessary for the first job?

Well, almost nothing is strictly necessary, given the great variety of jobs and potential qualifications. But there are some skills that I would highly recommend in most cases,  and I hope to detail this in a future post.

I’ve been learning my way around Lucene, the popular Java open source information retrieval software library, lately. Which is to say, I’ve been making mistakes with Lucene.

Much of the time spent with any new technology is in making mistakes, discovering mistakes, and correcting mistakes. An inexperienced programmer would view these mistakes as impediments to progress. It’s far more sensible, and more encouraging, to view these mistakes as the evidence of progress.

With that in mind, I’ll have no shame in sharing my newbie Lucene blunders, with hopes of helping someone that hasn’t yet had the opportunity to fail in all of these ways.

#1: Using a database.

As a software developer, I tend to think that data doesn’t exist unless it is in a relational database. Of course, Lucene would index the data, but that index, like the index in the back of a book, would simply tell me where to find the data. Not true. With Lucene, you can choose for each indexed field whether the field should be stored in the index or not.

Obviously, the right way to store your data will depend on many things, but storing it in the index is the simplest option available. Not only does this mean that you don’t need to set up, connect to, and populate a database before beginning, but it streamlines the process of going from a text query to the resulting document.

#2: Storing numeric data in Field, not NumericField.

Lucene will be able to search on numeric data as text, so long as numbers aren’t removed during the indexing process, regardless of whether you add it as a Field

doc.add(new Field("chapter", String.valueOf(chapter), Field.Store.YES,
                  Field.Index.NOT_ANALYZED));

or as a NumericField

doc.add(new NumericField("chapter", Field.Store.YES, false).setIntValue(chapter));

The big difference comes in the ability to perform a query on a range of values. Which can be nice.

#3: Leaving the IndexWriter open.

We all know that it’s important to close resources (files, DB connections, etc.) that are no longer needed. But you can usually get away with being a little sloppy when you are working on a toy project at home with one user. Not so with the IndexWriter.

I indexed the documents that I was hoping to search, failed to call close() on the writer, and found the following files in the index directory:

_0.fdt
_0.fdx
write.lock

I should have guessed what this meant, but I didn’t, and when I tried to search, I received the following:

org.apache.lucene.index.IndexNotFoundException: no segments* file found in
    org.apache.lucene.store.MMapDirectory@/home/user/path/to/index
    lockFactory=org.apache.lucene.store.NativeFSLockFactory@67e8a1f6: files: [write.lock, _0.fdx, _0.fdt]
  at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:712)
  at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:75)
  at org.apache.lucene.index.IndexReader.open(IndexReader.java:462)
  ...

which clues me in to what’s going on. The IndexWriter that I had long since forgotten about still owned the lock on the index. If I remember to close the writer, then the index directory has no write.lock file, and does have segments_1 and segments.gen files. More importantly, my queries no longer give the above stack trace.

#4: Setting text field to NOT_ANALYZED

In following an example, I used the following to add a free text field to a Document:

doc.add(new Field("text", text, Field.Store.YES,
                  Field.Index.NOT_ANALYZED));

I didn’t give much though to the NOT_ANALYZED, as I didn’t really care about analysis at this point … or so I thought.

But I did care about non of my queries finding anything. It’s pretty simple. NOT_ANALYZED means that the text is not broken down into individual tokens. In this case it can only match exact queries, which only makes sense for fields containing very few words, or fields that you don’t care to query. For any significant text that you want to be searchable, use ANALYZED.

#5: Thinking that TopDocs contained my documents

So I understood that my query, when executed, returned a TopDocs object. I’ll return that to the caller, who can then dig in and find the desired fields, right? Wrong. TopDocs does contain a ScoreDocs array, each of which contains an id for the relevant document. But to use that id in finding anything else about the document, you need the IndexSearcher itself. So the class that interacted with the IndexSearcher should also have the responsibility of returning something more understandable, perhaps a list of domain objects.

Your mistakes?

I hope someone finds value in my first mistakes. What newbie Lucene mistakes have you made?