YARN Command Line Logs

The simplified YARN log management can be a life saver. Being able to do yarn application and yarn logs is certainly handy if you do a lot of debugging and development. The alternative is that horrible web UI.

Initially I came across this problem retrieving logs from YARN according to the documentation. Let us say you have an application with the id application_1401296437855_0024. Trying a yarn logs -applicationId application_1401296437855_0024 left me as a question mark with the error:

14/05/29 03:24:54 INFO client.RMProxy: Connecting to ResourceManager at sandbox.hortonworks.com/192.168.109.138:8050
Logs not available at /app-logs/root/logs/application_1401296437855_0024
Log aggregation has not completed or is not enabled.

Note that the logs is obviously stored in HDFS, and a hdfs dfs -ls /app-logs/ shows that the logs are stored under /app-logs/hdfs. That meaning the log request must be issued as the hdfs user.

So that's basically it. You can find more information about YARN log management on the Hortonworks blog.

Tommy

Tommy is an analyst and incident handler with more than seven years of experience from the government and private industry. He holds an M.Sc. in Digital Forensics and a B.Tech. in information security