/ hadoop

YARN Command Line Logs

The simplified YARN log management can be a life saver. Being able to do yarn application and yarn logs is certainly handy if you do a lot of debugging and development. The alternative is that horrible web UI.

Initially I came across this problem retrieving logs from YARN according to the documentation. Let us say you have an application with the id application_1401296437855_0024. Trying a yarn logs -applicationId application_1401296437855_0024 left me as a question mark with the error:

14/05/29 03:24:54 INFO client.RMProxy: Connecting to ResourceManager at sandbox.hortonworks.com/192.168.109.138:8050
Logs not available at /app-logs/root/logs/application_1401296437855_0024
Log aggregation has not completed or is not enabled.

Note that the logs is obviously stored in HDFS, and a hdfs dfs -ls /app-logs/ shows that the logs are stored under /app-logs/hdfs. That meaning the log request must be issued as the hdfs user.

So that's basically it. You can find more information about YARN log management on the Hortonworks blog.