To configure logging of ALG trace is not enough to just set traceoptions section in [security alg]...
You should configure only required ALG module and security traceoptions like in example below:
[edit security traceoptions]
set file STrace size 10m files 3 world-readable
set flag all
[edit security alg sip traceoptions]
set flag all detail
Be advised that current settings could lead to high device load and logfile sizes. Be careful when choosing logfile parameters. In this example number of log files is limited to 3 and maximum file size is set to 10 megabytes.
You can now view captured logs by running in view mode:
show log STrace
