STR #3458: GLUT compatibility mode segfaults when there's no current window

STR #3458

Application:	FLTK Library
Status:	1 - Closed w/Resolution
Priority:	3 - Moderate, e.g. unable to compile the software
Scope:	3 - Applies to all machines and operating systems
Subsystem:	GLUT Emulation
Summary:	GLUT compatibility mode segfaults when there's no current window
Version:	1.3-current
Created By:	bschack
Assigned To:	AlbrechtS
Fix Version:	1.4.0
Fix Commit:	1fd6f0dd3af3183c0b41b94eae1379f8e60f3d96
Update Notification:	Receive EMails Don't Receive EMails

Trouble Report Files:

#1	bschack 20:35 Mar 31, 2018

foo.cxx
1k

#2	bschack 18:35 Apr 03, 2018

big.cxx
3k

Trouble Report Comments:

#1	bschack 19:20 Mar 30, 2018

When using GLUT compatibility mode, it's possible for there to be no current window - for example, when multiple windows have been created and the current one is deleted. In that case, FLTK correctly sets the current window to NULL. However, many functions (eg, glutGetWindow) always assume there's a current window, and in that case will try to dereference a NULL pointer, crashing as a result. GLUT (and presumably FreeGLUT), handle such cases more gracefully. For example, calling GLUT's glutGetWindow() with no current window returns 0. Below is a trivial piece of code that illustrates some of the problems. It's not very realistic, since it doesn't create any windows, but it's much more concise than creating multiple windows, with the same effects. int main(int argc, char *argv) { // GLUT - prints 0 // FLTK - segmentation fault printf("%d\n", glutGetWindow()); // GLUT - NOP // FLTK - segmentation fault glutSetWindowTitle("hello"); // GLUT - produces warning // FLTK - segmentation fault glutSetWindow(7); return 0; }

#2	AlbrechtS 01:20 Mar 31, 2018

Thanks for the report and the test code. Can you please provide full sample code that can be compiled and linked, preferable with build instructions (use "Post File")? This would help to test and fix FLTK's code to handle this case. TIA.

#3	bschack 20:36 Mar 31, 2018

Added a complete file - let me know if you have any questions or problems with it. Note that the GLUT stuff is written for OS X - for other systems the incantation will have to change, but I only have access to an Apple.

#4	AlbrechtS 10:32 Apr 02, 2018

Thanks for the full sample program. I could compile and run it on Linux with both the FLTK glut emulation and FreeGlut. I'm looking further into it... I modified the test program slightly. Here is an example output: glutSetWindow(7) freeglut ERROR: Function <glutSetWindow> called without first calling 'glutInit'. So where you wrote "GLUT - produces warning" this is an ERROR (program is terminated) in FreeGlut. In this particular case the error message is indeed confusing. The same kind of error would (should) happen if the index (7) is wrong (but 'glutInit' *was* called previously). I'm not sure what errors we can handle more gracefully (and what error message we should generate). I tested error handling with Fl::fatal() like so: glutSetWindow(7) FLTK ERROR: glutSetWindow() called without current glut window. At this time I'm not sure what Fl::fatal() would exactly do an other platforms, namely on Windows, but I'll check this. Would such an error handling be okay for you?

#5	bschack 18:35 Apr 03, 2018

I wrote an expanded program to test more functions, and compiled it on 3 systems: (1) GLUT version 3.7 (was packaged with OS X; now deprecated) (2) FreeGLUT 3.0.0 (3) FLTK's GLUT compatibility mode (FLTK 1.3.4) In broad terms, I would characterize their error-handling philosophies as: (1) Quietly ignore what can be ignored (2) Issue errors and quit (3) Crash As for my preference, I tend to like the GLUT philosophy, but I can see there being an argument for the FreeGLUT philosophy as well. As long as FLTK is consistent, I'm happy with either. I'm not thrilled with FLTK's current philosophy. :-) I should note that the Mac implementation of FreeGLUT seems unstable - sometimes a program will run without complaint one time, then produce a bunch of errors and fail the next, with no changes or recompilation in between. So, I wouldn't completely trust my FreeGLUT test results. I think my general impression of FreeGLUT's philosophy is correct though. My expanded program (included as big.cxx) tests every function I could find in FLTK that uses glut_window without first checking if it's null. Remember that when glutDestroyWindow() is called with the current window, FLTK (correctly) sets glut_window to NULL. So, in my opinion, FLTK code *must* check for a NULL glut_window before dereferencing it. What we do when we find a NULL pointer is the question. As mentioned before, I'm happy with either GLUT's or FreeGLUT's philosophy, as long as we're consistent. Having said that, there is one function that needs to be changed, and in a specific way: glutGetWindow(). According to the documentation, it returns the identifier of the current window, zero otherwise. That means it should look like this: inline int glutGetWindow() { if (glut_window) { return glut_window->number; } else { return 0; } } I hope this helps!

#6	AlbrechtS 07:51 Jan 01, 2019

Reduced priority to 3 (moderate). This issue should not block the release of FLTK 1.3.5.

#7	AlbrechtS 08:26 Oct 07, 2023

Fixed in Git repository. This is fixed in Git 'master' (FLTK 1.4.0), sorry for the long delay. Commit: 1fd6f0dd3af3183c0b41b94eae1379f8e60f3d96. I decided to use the "GLUT philosophy" to "quietly ignore what can be ignored". It appears that all function calls can be ignored, and those calls that return an 'int' value return 0 (zero). Everything else would IMHO have been much more work and is not necessary. If there are any issues with this fix please open a GitHub Issue (Bug Report) or report it in fltk.development. @bschack: many thanks for the test program (big.cxx). I modified it slightly and added it to our test programs (test/glut_test.cxx). Closing this STR now.