Thoughts, stories and ideas on cyber security

YARN Command Line Logs

The simplified YARN log management can be a life saver. Being able to do yarn application and yarn logs is certainly handy if you do a lot of debugging and development. The alternative is that horrible web UI.

Initially I came across this problem retrieving logs from YARN according to the documentation. Let us say you have an application with the id application_1401296437855_0024. Trying a yarn logs -applicationId application_1401296437855_0024 left me as a question mark with the error:

14/05/29 03:24:54 INFO client.RMProxy: Connecting to ResourceManager at
Logs not available at /app-logs/root/logs/application_1401296437855_0024
Log aggregation has not completed or is not enabled.

Note that the logs is obviously stored in HDFS, and a hdfs dfs -ls /app-logs/ shows that the logs are stored under /app-logs/hdfs. That meaning the log request must be issued as the hdfs user.

So that's basically it. You can find more information about YARN log management on the Hortonworks blog.


Tommy (B.Tech., M.Sc.) is a seasoned cyber security analyst with experience from both the government and private industry. He works daily with data- and intelligence-driven cyber security operations.