VisualGDB

VisualGDB is a third party plugin for Visual Studio, which can be downloaded under Tools->Extensions and Updates ... in the online Visual Studio Gallery.
You have to buy a license after a free trial period.
Official website.

With VisualGDB you can debug an App directly within Visual Studio while it's running on your App Platform.

Prerequisites

running App Platform (either on an ARM device of innovaphone or a virtual machine)
an App project

Installation

install VisualGDB
start Visual Studio
go to Tools -> VisualGDB -> SSH Host Manager
in SSH Connections add the connections you need by setting the IP address, the user name and the password (default is appname/appname)
under Host aliases add two aliases

appname-x86_64 (select as target an SSH Connection that runs under a virtual machine). This is for debugging x86_64 code.
appname-arm (select as target an SSH Connection that runs on an arm platform). This is for debugging arm code.

App Platform manager

Open your manager app inside your app platform, go to Settings and enable the developer mode. Save the change.
Reopen the same Settings again and enable the "Disable App security" checkmark.

This means, that the user, which will be created for each (newly) installed app will be able to login with appname/appname as user/password. This is needed for VisualGDB.
You have to uninstall and reinstall apps installed while this flag was active if you want to restore the app security!

If you disable the developer mode, the app security is reenabled automatically for new apps!

Usage

You can now start debugging by selecting one of the two solution configurations with VisualGDB support (arm-visual-gdb or x86_64-visual-gdb).
The debugging itself now works like within any other Visual Studio project.
More experienced users can use the VisualGDB settings menu to change configuration settings.

Opening core dumps

If one of your Apps crashed, a core dump will be automatically created inside /var/log/core_dumps/appname.
If you have the binary with debug information which caused this core dump, you can open it directly with Quick Debug,
which will jump to the code line which caused the crash and you can examine the stack trace and the variables (Debug->Quick debug with GDB in VS).

If you have a core dump from another app platform, copy it to your PC.
Connect to your local app platform with WinSCP (arm core dumps to arm app platform and x86_64 to x86_64 app platform!).
Copy the core dump to /home/admin
Copy the app binary and its debug file to /home/admin. The binary must be the binary which caused the crash!
Open the App folder in Vault -> Show History. Tick View folder history by version.
Select the day of the build under Dates & Labels under On or after and click Ok.
Get the tree on the label of the build recursivly to have sourcecode which matches.
Create an SSH host connection to your local machine with the user admin and the password ipapps if you do not have one yet.
In the quick debug settings, select this SSH host connection as linux system.
Select /home/admin/app-binary as debugged program.
Select /home/admin/coredump as coredump
Start ...

Generating binary with debug symbols from a release build

The debug symbols have been stripped from the release build but are stored in a separate file within your build directory, e.g. arm/appname/appname.debug.
Upload the debug symbol file with WinSCP to the app directory on your App Platform and execute the following command within an SSH client:

cat /apps/appname/appname.debug >> /apps/appname/appname

Attach to a running process

You can also attach to a process which is already running on the App Platform. You can see the required settings for Build->Attach to process here:
Note that you may have to enter the root password although the password for the admin user is requested.

Known Issues

Debugging Failed: In that case make sure, that you have created the directory for the application (generally /apps/app-name/app-name), and that this directory belongs to the user and group of the app (the appname). If not, create it and change the group and the user calling chgrp

Logging

Inside Apps we support an instance related logging mechanism. Therefor you use the Log function of your class which is derived from the base class AppInstance.
For more information take a look at debug.htm.

Manually opening a core dump on the LAP directly

You can open a core dump directly on the LAP with an SSH client.
Login with the admin user and su to root.

gdb /apps/appname/appname /var/log/core_dumps/appname/appname.core.dump # this opens a GDB shell
bt # this shows the stack trace
thread apply all bt # this shows the stack trace for all threads
Quit # quits GDB shell

Valgrind for leak checks

valgrind is a tool which can be used to detect memory leaks or errors (e.g. buffer overflows) within your App.
You can use it within an SSH client directly on your App Platform:

chmod +x /apps/appname/appname
valgrind --leak-check=full --show-leak-kinds=all --error-exitcode=1 /apps/appname/appname

Now the app is running and you can perform some operations, e.g. in the web UI.
Afterwards you stop the app by pressing CTRL+C inside the shell, which will trigger the termination of the app.
This may fail if you do not cleanup your app correctly ... In this case the app will hang and valgrind won't terminate.

If you have a leak, this may produce output like this, if you e.g. forgot to delete the debug class in your main.cpp:

        ==1506== HEAP SUMMARY:
        ==1506==     in use at exit: 20,601 bytes in 4 blocks
        ==1506==   total heap usage: 422 allocs, 418 frees, 174,870 bytes allocated
        ==1506==
        ==1506== 9 bytes in 1 blocks are still reachable in loss record 1 of 4
        ==1506==    at 0x4835474: malloc (vg_replace_malloc.c:299)
        ==1506==    by 0x4B2A8D3: strdup (in /lib/libc-2.22.so)
        ==1506==
        ==1506== 16 bytes in 1 blocks are still reachable in loss record 2 of 4
        ==1506==    at 0x4835C58: operator new(unsigned int) (vg_replace_malloc.c:328)
        ==1506==    by 0x2AF8B: IDebug::Create() (debug.cpp:371)
        ==1506==    by 0x2AFF3: __static_initialization_and_destruction_0(int, int) (debug.cpp:51)
        ==1506==    by 0x2B02B: _GLOBAL__sub_I_debug (debug.cpp:372)
        ==1506==    by 0x5961F: __libc_csu_init (in /mnt/sda2/apps/sysadmin)
        ==1506==
        ==1506== 352 bytes in 1 blocks are still reachable in loss record 3 of 4
        ==1506==    at 0x4835474: malloc (vg_replace_malloc.c:299)
        ==1506==    by 0x4B1021F: ??? (in /lib/libc-2.22.so)
        ==1506==    by 0x2A61F: Debug::OpenFile() (debug.cpp:208)
        ==1506==    by 0x2990B: Debug::Debug() (debug.cpp:69)
        ==1506==    by 0x2AF9B: IDebug::Create() (debug.cpp:371)
        ==1506==    by 0x2AFF3: __static_initialization_and_destruction_0(int, int) (debug.cpp:51)
        ==1506==    by 0x2B02B: _GLOBAL__sub_I_debug (debug.cpp:372)
        ==1506==    by 0x5961F: __libc_csu_init (in /mnt/sda2/apps/sysadmin)
        ==1506==
        ==1506== 20,224 bytes in 1 blocks are still reachable in loss record 4 of 4
        ==1506==    at 0x4835474: malloc (vg_replace_malloc.c:299)
        ==1506==    by 0x490EE2F: ??? (in /usr/lib/libstdc++.so.6.0.21)
        ==1506==
        ==1506== LEAK SUMMARY:
        ==1506==    definitely lost: 0 bytes in 0 blocks
        ==1506==    indirectly lost: 0 bytes in 0 blocks
        ==1506==      possibly lost: 0 bytes in 0 blocks
        ==1506==    still reachable: 20,601 bytes in 4 blocks
        ==1506==         suppressed: 0 bytes in 0 blocks
        ==1506==
        ==1506== For counts of detected and suppressed errors, rerun with: -v
        ==1506== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Known Issues

still reachable leak from /usr/lib/libstdc++.so.6.0.21: This is not a real leak, but a buffer which has been allocated within the standard library. You may ignore such leaks.
See the valgrind FAQ for more information.