Troubleshooting Rundeck Remote Resource Issues

rundeck-logoAs part of an ongoing effort to migrate a significant part of our stack into the cloud, we recently moved our job scheduling system, Rundeck, from an on-premise bare metal instance to AWS. With the exception of a few minor annoyances–from an operational standpoint–pretty much everything worked. It didn’t take long, however, to notice jobs weren’t running on-time. Minutely jobs were running between 30-50 seconds off, skipping minutes, etc. As we had a near-identical configuration, we weren’t really sure what the issue was. Given the lack of documentation for Rundeck architecture and having seen similar unanswered questions across various message boards, after three of us had spent a couple days on the issue, we wanted to post the steps we took to troubleshoot and resolve the issue.

Job Concurrency Limitations
As the most common cause of items not running on-time in a scheduling system is related to the permitted level of concurrency, and we saw no CPU-related run queue issues, we first expected the delay to be limited by the number of available worker threads. So, per the Rundeck Administration tuning guide, we increased the Quartz scheduler’s thread pool size (org.quartz.threadPool.threadCount), related JVM heap settings, and restarted the service. We noticed zero effect on jobs and verified the thread pool was operating as expected.

AWS Instance Sizing/Performance
At this point, we knew the Quartz scheduler was not the issue. As everything else in the configuration was identical, while we hadn’t seen any CPU-related issues, we figured it could just be related virtualization slowness. So, just to rule it out, we rolled the next largest box. As expected, given we had seen nothing to indicate CPU issues, we saw no difference in performance on a larger instance; jobs were still running 30-50 seconds behind.

Java Version Differences
After comparing packages between our on-premise box and AWS box, we found our on-prem box to be running Java 1.8.0, while our AWS instance was running 1.7.0. We upgraded the AWS instance to 1.8.0 and noticed a five-or-so-second improvement. The UI was much faster. But no huge improvement in runtime.

Waiting for Execution
During one of the times we launched a job manually, we noticed it became queued immediately, but sat there in the UI for over 30s saying, “Waiting for execution to start.” We thought this was odd, as the box wasn’t exhibiting any CPU-related issues and there were hundreds of free threads in the scheduler to execute the job. We searched and came across this thread, which seems very similar and was unanswered.

Local vs SSH Job
In an effort to make sure SSH wasn’t the issue, we created a local job that simply echoed the current timestamp into a file in /tmp. This job too ran between 30-50 seconds behind the time it was supposed to start. Obviously, this isn’t an SSH issue.

MySQL Optimization
As we’re using MySQL as the backend of Rundeck, we figured there may be something up with database performance. After taking a look, we noticed several queries taking several seconds to complete. Accordingly, we realized the tables weren’t indexed properly for those queries. Rather than creating them ourselves, we searched and found a ticketed performance issue specifically related to missing MySQL indices. We created these indices and, while we no longer saw any slow queries, we saw no improvement in job execution.

Reducing the Number of Concurrent Jobs
To see whether this issue occurred with the same frequency with fewer jobs, we disabled all but one minutely-executed job. While it was still off by ~6s, it was far better than 30-50s. As we enabled other jobs one-by-one, we saw increasing start delays. As it was still 6s off, we figured something else must still be going on.

Watching the Filesystem
During the time we had been watching the single job logs, we noticed a new job log was being created at EXACTLY the right time the job was supposed to start. That is, at the first second of each minute. Unfortunately, when we looked at all other logs, none indicated this job had even started. So, how did this file get created at the right time, but no scheduler log showed this job had actually yet started? Let’s eliminate all the variables.

Eliminating Outside Variables
Our Rundeck configuration includes multiple servers, which have nodes resourced from a local XML file (resources.source.N.config.file) as well as multiple servers via a URL in the project resource configuration (resources.source.N.config.url). We disabled all URL-based ones and noticed our job ran perfectly on-time over and over again. So, something was up with the URL-based configs. We noticed, for some reason, caching was disabled (resources.source.N.config.cache=false), so we set that to true, reenabled our URL-based resources, and restarted. Same old performance issues. Looking at the code, that’s only because this adds additional cache headers to the HTTP request. As everyone who knows HTTP knows, dynamic endpoints still have to do all the work, even if they don’t return any results. So, we tried to fetch that resource directly via cURL. It took ~2s to fetch the resource file from each server. Yikers! We then researched the resources.source.N.config.timeout option, only to find it wouldn’t really help in this case. Lastly, we decided to try saving those resource files locally and using them directly. After that, all jobs ran perfectly on time.

So, when a job gets kicked off by Rundeck, it appears it’s not officially “started” until after the resource configuration is fetched. As multiple jobs run per minute, each one fetching N number of URL-based resources, this caused a stampeding herd to the other Rundeck servers and, accordingly, a significant lag in execution. This also explains why–in the more-jobs-per-minute case–we saw greater lag with each added job.

Update: Response from Rundeck

The URL source is a blocking call so unfortunately it can lead to this kind of problem. We have discussed the idea to enable an asynchronous URL request to help avoid this kind of lag problem.

With the exception of making calls to multiple URL sources concurrently, thereby reducing startup-time from the sum of all response times to the maximum response time, I’m not sure how this could be improved in any asynchronous manner. As all nodes are required at the start of a job–to me–node retrieval must be remain synchronous, but it could be improved with smarter caching on the HTTP server. ¯\_(ツ)_/¯

Reflecting on Node 2015

mohonk-sceneryAfter an information-fueled three days, our team has returned from Mohonk Mountain House, where we held our sixth annual internal engineering conference, Node.

While Node started out as a development-only retreat, it has since become an engineering-wide conference consisting of our development, data science, product, quality assurance, and operations teams. As everyone in the technology industry has experienced at one time or another, it’s hard to find time to get to know peers, learn new things, and stay on top of organizational objectives when one is heads-down on a project; this is why we created Node.

Node is designed to provide our organization with a dedicated event, away from the office, to share knowledge, learn something new, and get to know each other in a collaborative environment. The first, knowledge-sharing, is accomplished via conference-like sessions and workshops presented by our own internal colleagues.

This year, we had a number of internal, team-oriented, tracks. Within each track was a presentation or workshop related to that technology or team. As an example, Corky Brown and Bill Sykes gave a kick-ass workshop on development with our new client-side rendered web code, Phoenix. Likewise, our new Director of Data Science, Anton Slutsky, Ph.D., presented, “Data Science @ MeetMe,” a session designed to give everyone a basic understanding of Data Science, Big Data, its importance, and its role in our organization. In addition to learning more about the work other teams are doing or how to use some of our new tools, we also bring in external speakers, who provide us with experience and insight outside of our day-to-day work.

seb-imgAs Node is composed of individuals with varying degrees of technical depth, we try our best to attract external speakers who provide experience and content beneficial to everyone. Oftentimes, these speakers are aligned with either a task or a technology we’re working on, or one we plan to put into effect in the coming year. Given the significant effort our engineering team has put into Cucumber this year, we felt it imperative to spend a considerable amount of time focused on quality. For that reason, we invited Seb Rose, co-author of The Cucumber for Java Book, to spend two days with our team.

Seb covered agile development concepts, BDD/TDD, and detailed aspects of feature/scenario development with Cucumber. Unlike other presenters, who lecture for hours on end, Seb’s group-oriented hands-on approach was well-received and left many with a better understanding of not only the concepts surrounding Cucumber, but also the best ways to use the technology itself.

Alternatively, for those who wanted something a little more technical, we asked William Kennedy, author of Go in Action, to give us an overview. Bill crammed two days worth of Go programming language training into only eight hours. To accomplish this, we skipped the hands-on examples.

node-2016-dinner2In addition to the content, we spent the remaining hours sleeping or in spontaneous team-building activities. Whether it was sitting together watching the Philadelphia Eagles vs. Dallas Cowboys game, hanging out at lunch, laughing at dinner, sightseeing, group photography, relaxing in the lounge, playing Heads Up!, or participating in a highly-competitive game of Texas Hold’em, we spent a good amount of time getting to know and collaborate with each other. This was especially useful for our new team members.

Lastly, every Node is a learning experience and improving it is an iterative process. What did we learn this year? For one, programming language trainings should be less intense. Some thought the Go training pace was fine, while others felt it was a bit too rushed. Likewise, everyone wants a little time after lunch to explore the venue we’re at. We’ll be sure to take that into account next year.

All in all, it was another great experience for the team. I’d like to thank everyone who attended, presented, and helped make it happen. Specifically, on the engineering side, with input and feedback from multiple teams, Node was planned primarily by our own DevOps Lead Architect, Jason Lotito, who did an outstanding job. With everyone reinvigorated, on the same page, and understanding what our 2016 goals are, I’m excited to see what we will accomplish!

A look at Apple’s new programming language: Swift

There has been a lot of buzz over Apple’s new programming language, Swift, and many questions as well. Why did Apple need a new language? How is it different from Objective-C? As a developer, should I learn Swift? We’ll dive into these topics in this post.

A Brief History

Swift is the heir to Objective-C, so it’s very difficult to talk about Swift without first talking in detail about Objective-C. Objective-C was created in the early 1980s and descended from both C and the Smalltalk Language.

While most every programmer is familiar with C, Smaltalk is more obscure. It’s an object-oriented language and uses a message-passing style to call methods. (This will come into play later when we discuss Swift syntax.)

During his exile from Apple, Steve Jobs formed a company called NeXT. This company licensed Objective-C in 1988, and they used the language to build an operating system called NextStep. Flash forward to 1996, Apple’s in bit of a rut and they’re looking to update their operating system…Apple acquires NeXT and uses it as the basis of Mac OS X.

Things were fine for a few years and a small amount of developers quietly made Mac desktop applications. In 2007, everything changed when the iPhone was released. Developers were soon able to release third party apps in the App Store which led to an explosion of interest in Objective-C.

What’s wrong with Objective-C?

So what’s the problem? Why can’t we just use Objective-C until the end of time?
In short, Objective-C was never intended to be as popular as it became. It was used for the OS and Mac apps, but there wasn’t a mobile app industry at the time and no one could anticipate what a game changer the iPhone would be.

Objective-C has a somewhat tough barrier to entry because it doesn’t look like a lot of the languages that descended from C like C++, Java or C#. As far as aesthetics go, Objective-C is fairly hideous.

Here’s a sample of Objective-C code:

For comparison, this is how you could accomplish the same thing in Java:

Objective-C can be intimidating to programmers who aren’t used to this syntax. Let’s take a look at some of the issues here.

  • We have square brackets all over there place
  • An asterisk before the variable name
  • @ symbols
  • Lots of extra words cluttering up like alloc and initWithFormat
  • What’s with this “NS” and why is that cluttering up the String?

You’ll see this “NS” in many classes in Objective-C: NSArray, NSString, NSDictionary, etc. It’s a nightmare for autocomplete.

Because Objective-C doesn’t have a concept of namespaces, class names have to be unique, so you need to add a prefix like “NS”. If we were to create a custom string in the MeetMe app, it might have a name like MMString.

The prefix “NS” is a relic from when Objective-C and its Foundation framework were built using NextStep. Again, this was created way before anyone could anticipate how popular it would be to code in the language. If they had a crystal ball and saw the future, there’s a good chance the team at NeXT would have made the entire language easier to use.

Over the years, Objective-C has gotten friendlier with more features you’d expect out of a programming language (like blocks and literals), but Apple was looking for a lighter language with less barrier to entry and to clean up some of the relics of the past — Swift is their answer.


One of the main benefits of Swift is that it is a much more concise language than Objective-C. Gone are:

  • Semicolons ;
  • [Square brackets ]
  • #import statements for classes
  • Alloc/init, lengthy method names

Back to the previous code samples with the strings — here is the same line in Swift, and you can see it has done away with all the weirdness we pointed out in the previous example, looking a lot more like a modern language:

You can also overload the arithmetic operators with your own custom functionality. In Objective-C, there is no overloading; you have to create a new method name if you want to have different types of parameters for the same method. As you may know, autocomplete gets pretty cluttered with all these extra methods.

In Objective-C you need to access properties with the “self” keyword and also call methods on self. In Swift, the self is implied in properties; you can write it out if you want to be explicit, but you can omit self for conciseness. If you call a function without specifying a variable to call it on, the function is implied to call on self so you don’t need to declare that either.



This is an example they used in the Apple Keynote when they first introduced Swift.

Here, you see some code you’d commonly see in an Objective-C app where you spend a lot of time checking if the delegate is nil and checking if the delegate has a particular method before you call it.

In Swift, they’ve cut down on a lot of typing with Optionals:

The use of Optionals is indicated by the question marks. (Optionals will come up later when I discuss safety.)

Another change that results in more concise code is the use of strict types in collections.

Here is a sample method from Objective-C:

You pass in an array of donuts, but the compiler doesn’t know that the array only contains NSDonuts. We know it because of clues from the method name (and maybe the documentation) but, for all the compiler knows, there could be strings in that donut array.

So, we create a donut variable and cast it as NSDonut to assure the compiler that it’s ok to call eatDonutMethod on it. Is there a better way? Here’s how this same method could be implemented in Swift:

The syntax is a little flipped, but we have the same name of the array — donuts. Instead of a general array type, this has an array strictly of donuts. This allows us to call donut methods on any object in the array without needing to cast it because it has to be a donut; the code won’t compile if you try to send an array of strings to this method.

It’s a great win for more concise code, but this also demonstrates another feature of Swift — its emphasis on safety!


All variables are statically typed in Swift, meaning you have to declare if a variable is an Int or a String; for example, you can’t just say it’s a variable and call any method you want on it later.

While Objective-C is also statically typed, it uses the generic id property in collections as we previously discussed. This sometimes leads to errors happening at runtime when a collection ends up having a type that you weren’t expecting. In Swift, because collections are of a particular type, there will be an error before you compile rather than at runtime if you do pass in the wrong type.

Type can also be inferred; as you see in this example, you don’t need to explicitly declare the variable type.

Here, it is inferred that donutArray is an array of donuts based on the assignment. Under the hood, this is still statically typed; donutArray is an array of Donuts, even though you didn’t have to explicitly type it out like the line above. You get the safety of static typing, but it’s also concise.

In Swift, variables are only able to be changed if you declare them a var. The preferred practice is to declare your variables as constants using the ‘let’ keyword. Not only does it optimize the code and make it faster, it adds safety by throwing you a warning if you try to change the contents.

Swift really tries to get you to think about how you’re going to use each variable, which we also see with Optionals. Nil or null pointers cause a lot of problems in programming, and different languages have different ways of dealing with it. In a lot of languages, your program will crash if you try to call a method on a null object.

Because Objective-C was inspired by the SmallTalk language, it uses message passing to invoke methods (told you we would come back to that!). A feature of message passing is that you can get away with calling a method on a variable that doesn’t have the method and the compiler won’t complain. In Objective-C, messaging nil returns nil, so no harm, no foul, no crash.

And yet, this isn’t ideal. What if you need the return value from this method and getting back nil will crash your program? You could wrap this in a nil check, or use the respondsToSelector method, and that is what we have to do all the time in Objective-C. Swift is looking for a better way to handle the nil problem.

Enter Optionals. Here’s an example from earlier — let’s say later on you want to clear out the donuts and set the variable to nil:

You cannot do this because then donutArray would no longer be an array of donuts. A var expects a value at all points of its life cycle, from initialization until it is deallocated. How you can accomplish this is by making donutArray an Optional:

An Optional behaves like how variables in Objective-C behave in that they can be nil at any time. They also do not need to be initialized to a value.

Optionals should be used only when necessary because they work around the safety Swift has created to try to eliminate runtime errors that crop up around surprise nil variables. The preferred order to create variables is:

  1. let constant
  2. var if it needs to change
  3. optional var if it needs to be nil at some point

Optionals are generally meant to be used with a question mark, meaning that there may be an object there or it may be nil. Sometimes you need to guarantee that there’s a value there. The easiest was to use the underlying value of an option (and least safe) is to to force unwrap it by using an exclamation point.

This is a brute force way of saying, “Just trust me, there’s a value there.” This should be avoided unless absolutely necessary because it goes around the safety of optionals and the program will crash if it turns out you were wrong and there’s no object there.

The preferred way to handle this is using a new statement, “if let” (also known as Optional Binding). This creates a new local constant equal to your optional only if there’s a value there. You then safely use the new constant inside the block.

Optional binding is similar to a nil check, so that seems to go against Swift’s push for concise code. I think that, as Swift develops, the syntax for this situation will become more concise. For now, this is the way to safely handle optionals when you need to guarantee they’re non-nil.

Other Safety Features

Swift has eliminated several features that allowed programmers to be a little lazy at the expense of safety. Here are two conditional statements in Objective-C that both do the same thing:

If you only have one line inside the body of an if statement, you have the option of omitting the curly braces in Objective-C. The problem is that it’s really easy to think that you’re just going to need one line, then come back to your code sometime later and add more lines not realizing that you also needed to add braces. Those extra lines will not be tested by the conditional statement; they will always execute!

This is an insidious error because the compiler will not complain; it just always executes the extra code. You may not even know there’s a problem until months or years later when unusual things happen. The most infamous recent example of this is Apple’s “goto fail” SSL bug, which happened while using this same shortcut:

Apple decided that this convenience was just not worth the risk, so the brace-less conditional checking in Swift is disallowed. Another area Apple made safer while actually letting us be lazier is in switch statements. Here is an example switch statement in Swift:

Notice there are no “break” statements anywhere. They are no longer an option in Swift — every case breaks automatically. While this takes away some of the flexibility of being able to fall through to the next case, it eliminates the common problem of forgetting to type “break” after every case and accidentally falling through when you didn’t intend to.

Instead of falling through to additional cases, Swift allows adding several options to the same case (see the “left”, “right” case). And, thankfully, you can now use non-number types inside switch statements such as strings (as you see in the above example), classes, and optionals.

Finally, switch statements must now be exhaustive, which usually means that a default statement is required. In the case of an enum you can usually evaluate all cases; however, there is no way you can evaluate every possible string combination, so use default if you don’t want to get a compiler error!

There are lots of other examples of safety in Swift, too many to go through in this post. The main takeaway is that Swift was designed to be easier to learn, faster to run, and safer to code in than Objective-C. It is evolving and has had some significant changes in its short lifetime. It remains to be seen how quickly Swift will supplant Objective-C, but there seems to be justified enthusiasm for non-Objective-C programmers looking to get into iOS app development.

Hackd project – Remote Car Starter

It’s HACKD season again at MeetMe! For those of you that are not familiar with the concept, hackathons are events that some companies, schools, and other organizations put together where people work on the projects they want, like new technologies, inventions, ideas, etc., that they usually doesn’t have time to spend on. I’ve had a project on the back burner for a while that I wanted to work on but, because lack of experience and knowledge (a.k.a self confidence), I never went forward with it.

MeetMe offers HACKD every couple of months and during one of the recent events, I decided to finally work on this project, which I call “Really Remote Starter”.


About 3 years ago, I lived in Northwest Mexico, near Arizona, US. Since then I moved to the US East Coast. So I basically moved from here:


…to here:


I found out that in this weather I need to sit in my car for a while until it gets warm. Sitting in the car does not provide an ideal comfort level. Even if you turn the heat on, it’s useless for those first few minutes.


By the end of the first winter season, I desperately wanted a remote starter for my car. I thought, “it would be nice to start my car just from the warmth of my home or office and then just go to my car into a warm interior and go”. Then, when I got a remote starter for my car, I quickly realized that it wasn’t always going to be that useful.

At work, the parking lot is located at the back of the business center, and it fills pretty quickly. It can take you a good 5 to 10 minutes to walk to your car. Also, in other cases like parking at malls, movie theaters, etc. makes the Remote Starter less useful. Why? Remote starters and car alarms have RF remotes, which are very limited in terms of reach.

In my other car, in Mexico, I have a high end car alarm with a two-way RF remote that promises a one mile reach. Well it has a one mile reach in a place like this:


If you start adding cars, walls, and other barriers, the reach decreases dramatically. So, I came up with the idea to make my remote starter to start from wherever, without a distance restriction.


I wanted to start my car from wherever with my cellphone, sending the car some kind of signal. I wasn’t sure exactly how to achieve this, so I did some research and looked at a bunch of Youtube videos and written tutorials. It seemed that I wasn’t the first one to try this. It seemed that everyone wanted to use bluetooth. It makes sense–your cellphone has bluetooth and you can use an arduino or a raspberry pi with a bluetooth module and voilá. But that wasn’t enough for me because that doesn’t solve my problem. Although bluetooth could have longer reach than RF, it still has that distance restriction. I wanted something that doesn’t care how far are you from your car. Then the WIFI idea came to me. “What if I put a WIFI module in the arduino or the raspberry pi?” Nope, that is not going to work either.

It should have WIFI coverage everywhere and there’s no WIFI coverage everywhere; you also have to deal with password-protected networks and stuff like that. Same happens with 3G or 4G–there’s no 100% 3G/4G coverage everywhere, and it will turn this to a very expensive solution since 3G/4G mobile plans are expensive. But, what if I send an SMS to my starter? Let’s think about it: cellular signal coverage is pretty much 100% (only remote places are deficient) and SMS can be received pretty much everywhere. Also, very basic, cheap plans can give you this advantage. Got it!

Then HACKD weekend was approaching. I researched a little bit about which board to use, there’s a bunch of options out there, not just the arduino and the raspberry pi, but settled on the arduino basically because it was the right price and the complexity level required.


After deciding what to use and how to use it, I got everything I thought I would need.


  • Arduino Uno Rev 3 (link)
  • GPRS Module (Geeetech SIMCOM SIM900 Quad-band GSM GPRS Shield ) (link)
  • Breadboard and Jumper wires (link)
  • Soldering Kit (link)
  • Multimeter (link)
  • Cables
  • Voltage Regulator (link)
  • Switch Relay (link)

* The breadboard, jumper wires, soldering kit and multimeter links are just for reference, you can get other model, styles, brands, etc. You also don’t need to buy anything from Amazon, you can get everything from ebay or electronics specialized stores.

* Voltage regulator. Arduino documentation says that it works with 5V to 20V, but it can became unstable under 7V and you can burn it over 12V. Common car voltage is 12V so I needed this to convert down the voltage to turn on the arduino. However after 2 or 3 days @ 5V it became unstable. I switched since then to a 9V Voltage Regulator and that solved the problem.

Well that’s pretty much everything I needed. Now the process.


Installing a remote starter to a modern car isn’t an easy task. There are now transponders in the keys that you have to know how to bypass, you need to know different signals for several different instruments of the car, and you also need some security restrictions, like make it so the car shuts off some time after being started without a key (you could forget it was started, for example) or make it so the car can’t be driven without a key. (A thief could take advantage of the remote starter to just jump in and drive.) Because all of that logic was already implemented on the remote starter my car already had and the hackathon was a 3 day event, I decided just to hang my stuff off the installation already there.

My plan then was that I would send an SMS from my cell to the arduino (received via the GPRS Module). Then the arduino would send a signal thru a cable to the remote starter via it’s remote control. All of this would be located on a hidden place inside the car. Here’s the diagram.


alarma1 (1)

So I took out the remote from it’s case and connected one cable on each pole of each button

IMG_20141008_182952 IMG-20141011-WA0009 IMG-20141011-WA0040

Then I connected one switch relay to each button on the switches I had on the COM and N.O. pins


Then, all three switches connected to all three buttons of the remote need to connect one of their left pins to the negative of the arduino and each of the left pins on each of the three switches have to be connected to the three 5v pins that we are going to use to send the signals to it. So at the end all connected looked like this:

IMG-20141011-WA0042 IMG-20141011-WA0045

This module has to be connected to a continuous power source from the car; each car is different so anyone else trying has to figure that out. I added a voltage regulator because the reasons I already stated above.


The language Arduino basically uses is C++. Arduino has an IDE that we can use, downloadable from here. I used it for this project and it helps and works fine. Then we also need a Serial Terminal Software to communicate with the Arduino and the Geeetech SIMCOM SIM900 Quad-band GSM GPRS Shield, to send and receive SMS messages. On the Shield documentation located here, they offer some different options for different platforms, I downloaded CoolTerm for my Mac OSX from here and it worked perfectly. There’s some setup you need to do in it to communicate with the Arduino and the Shield. Ah! You also need a SIM Card with a plan installed on the Geeetech Shield.

Terminal Pre-setup

  1. Connect to the computer USB port and turn on Arduino (with the Geeetech Shield mounted).
  2. Go to CoolTerm and click options.
  3. Pick port, usually usbmodemfd something, if doesn’t appear click Re-Scan Serial Ports at the bottom.
  4. Pick 19200 Baud Rate.
  5. Ready to send and receive commands and read them.

There’s a second “language” or set of commands to learn here, and those are the ones that are used to actually send and receive the SMS messages. There’s a whole bunch of commands, not only for SMS, there’s commands to make a call, to receive it, to store, delete, refresh SMS, and all SIM card activities, even internet browsing. That reference can be located here.

Arduino Code

That code can be also viewed here and is going to be running in a neverending loop in the arduino and basically waits for the SIM card to get an SMS. Once the SMS arrives, it sends a double pulse (needed by the remote buttons) to the desired remote button, depending of the content of the SMS received. After that it deletes all the messages to avoid filling up the SIM Card.

Well, after “uploading” that to our Arduino, we can manually send SMS messages to our installation and it should work. This is a little video when I was testing it:

This is with the car already connected to the module:

I have to thank for this part of the code to my friends and coworkers Bob Dauth and Diana Shkolnikov, who helped me a lot with C++ because I had no C++ coding experience at all. After that, it was mostly done, however, I wanted an app that with a tap of a button I could tell my car to start or to open and close the door locks. I asked my also friend and coworker David Breneisen to help me with that. He came up in less than half hour with a very simple app that does the work. Here’s a screenshot:

Screenshot_2015-02-25-09-36-17Pretty much what it does is to send the SMS when you tap the button and that’s it. It also gets confirmation and shows a toast message when the SMS has been sent. Because its so many files you can find the code here. After putting that together, we finished our project.

I was able to use it all winter long and it was awesome!

Now, for this most recent HACKD, I wanted to improve on it. I projected the following changes:

  1. Remove the phone number from the code in the app and put a settings page so the user can change the phone number of the car.
  2. Get better-looking buttons with icons instead of just text.
  3. Get reporting back from the car when the car has been started, closed, or opened after the SMS is received.
  4. Add an alert to be sent by the car when a door has been opened.

This last one representing the biggest challenge because I needed to hook up more stuff to the car. I wanted to get reporting back from the car when it has started or closed or opened because there were times where my cell plan has expired and the SMS weren’t going out, I didn’t noticed that the car wasn’t started, and I was going out in the cold just to find out that the car hasn’t been started. Or vice-versa, the SMS was sent, but the car didn’t get the order, because the plan on the Arduino’s chip was expired.

For these new challenges I had to construct a new, more complex circuit represented in the following diagram:


I have to thank here to my friend from México, Carlos Pinto; he is an Electronics Engineer and helped me with coming up with this circuit. A little bit of description.

1. The icons on the top left corner are switches on the doors, this switches send the signal to the car computer to let it know a door has been opened, you can see an light coming off on your dashboard when a door is open, these are the responsibles.

We need to connect those sensors to the Arduino somehow to get the signal. There’s some considerations to make — these are negative signals, and we need diodes in order to avoid the signal to go between them.

2. To the center (Doors Relay), it’s basically a signal converter. That relay basically takes the negative signal from any of the doors and sends a positive signal to a voltage regulator.

3. The signal from the Doors Relay is converted to a 12V signal so that’s why we need a voltage regulator, to convert the 12V to a 5V signal that is going to be connected to an input pin on the Arduino (we convert it to 5V because that’s what the Arduino supports).

The rest of the circuit is pretty much the same as it was, just the connection from the car to power up the Arduino and the switches needed for sending the signals to the remote control of the starter. Then we need to update our code for the Arduino as follows:

This code is also on my Github and I have to thank Drew Denardo also friend and coworker for helping me with this code and putting together the new version of the Android app that now looks like this:

Screenshot_2015-04-22-17-56-40 Screenshot_2015-04-22-17-56-55

The code for this version can be also viewed here. After presenting and wrapping up the new version of the remote starter in the HACKD event, I realized an issue on the way it is programmed now; it sends me an SMS when a door has been opened or closed, even when I open or close the door. Obviously I don’t need an SMS telling me a door has been opened/closed if I’m the one doing it. I need to add some kind of logic to tell it when it is me doing or not. But I guess that’s going to be for the next HACKD?

Android: Determine when App is Opened or Closed

The Problem

Inevitably when developing an Android application there will come a time when you need to detect when it comes into the foreground, and when the user leaves.  Surprisingly, there is no easy way to do this! It isn’t hard to detect when the application is first opened, but its not so simple to determine when it is re-opened or closed.

This post will lay out a technique that can be used to detect when an application has been opened, re-opened, or closed.

Lets Get Started

At the core of determining if an application is opened or closed is if any of its activities are currently being displayed. So lets take a very simple example; an application that has exactly one Activity, and does not support landscape mode. All it would take to determine when the application opened or closed would be the Activity’s onStart and onStop methods.

The problem is this approach quickly breaks down as soon as you add landscape support.  If we rotate the device then this Activity will be re-created and onStart will run a second time resulting in incorrectly detecting the app was opened a second time.

In order to handle device rotation we need to add a validation step. This validation needs to launch a timer whenever the Activity stops in order to detect if we quickly see another Activity in the app start. If we do not, then the user has really exited the app, if we do then we know that the user is still within the app.

This validation also allows supporting an application with multiple Activities. Since transitioning from one Activity to another is handled by this validation as well.

So using this technique I created a manager class that all Activities report to when they become visible and are no longer visible. This manager class handles the validation step for each Activity to avoid accidental detection. It also utilizes a publisher-subscriber (Observer) pattern to allow any interested parties to be notified when the application opens or closes.

To use this manager there are three steps:

1) Add it to your codebase

2) Activities Must Notify of Visibility Changes

Have all of your Activities implement the following code to notify the manager when they become visible and when they are no longer visible. This is best achieved by adding this code to your base Activity class.

3) Subscribe to Be Notified of Foreground Changes

Subscribe to be notified of app foreground state changes from somewhere of interest. The application class onCreate method is a prime placement to ensure that you get notified every time the app enters or leaves the foreground.

Further thought

There are a few details that need further discussion.  These changes may need to be made in order to calibrate this for your particular application.

Validation Time

How long should this validation timer be that is used to detect that your app has really entered the background? In the code above it set to 30 seconds for a few reasons.

There are third party Activities that can appear full screen when the application is running. A few common examples of these are in Google In App Purchasing or Facebook signup. Both of these essentially send the application into the background since none of you applications Activities are in the foreground when those are displaying. It is not desirable for those to count as the user “leaving the app” since they really haven’t. The 30 second timeout helps to catch this case. For example if a user completes an in app purchase in under 30 seconds, which the vast majority of users will, then they will not be counted accidentally as “leaving the app”.

If this situation doesn’t apply to you, then I recommend setting your validation time to around four seconds. That is enough of a safety margin for slow devices to create the next Activity when rotating.

CPU Sleep

A potential issue is there is no guarantee that the CPU will still be running long enough for the background detection to occur if the user locks their phone quickly after exiting the app (or locks the phone with the app still open).  In order to guarantee this always works as expected you would need to take a wake lock until the app closed event can be validated to prevent the CPU from sleeping. In practice however this did not seem to be a big enough issue to warrant use of a wake lock.

Determining How The App Was Launched

Great so now we know how to detect when the application was opened or closed, but we don’t know how the application was opened. Did the user click on a notification? Did they click on a link? Or did they just launch the app from the icon or recents tray?

Tracking Launch Mechanism

First we need somewhere to track how the application was opened. In this code I have added an enum to the application class to track how the application was opened. This builds upon the last example so we can print a log when the application is opened, along with how the application was opened.

Setting the Launch Mechanism

Great so now we can print off the launch mechanism when the app is opened, but we never actually set it! So the next step is to set it when the user opens the app via a link or a notification. If its not set to one of those two, then the user must have opened the app directly.

Tracking Link Clicks

To track the user clicking a link that caused the app to open find the place in your code where you intercept your links and place the following code to track the launch mechanism. Make sure this is called before your Activity’s onStart(). This could be needed in many places or just one common link intercepter depending on your architecture.

Tracking Notifications

Unfortunately tracking notifications is a bit trickier. Notifications are shown and when clicked a PendingIntent that you previously bound to them is opened. The trick here is to add a flag to all PendingIntents that you make for a notification that indicates that this Intent is an Intent from a Notification. In other words, when the intent eventually opens some Activity, we need to be able to detect that this intent was created from a Notification.

So for example when creating any PendingIntent for a notification add the following to each intent:

Now all we need to do is check for that flag in every single Activity (add to your base Activity class). If we detect the flag then we know this Activity was launched via a notification and we can set the launch mechanism to notification. This must be done in onCreate so that it is set prior to the application coming into the foreground (which will trigger the printing of the launch mechanism).

And there you have it! You can now detect not only when the app is opened or closed, but how it was opened as well.


Do you find this sort of work interesting? Do you love quaint Bucks County Pennsylvania? Are you an amazing Android Developer? Then we would love to hear from you!

Featured image by Dsimic – license via CC BY-SA 3.0