Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ThreadManager: Add simple priority queues #16812

Merged
merged 5 commits into from
Feb 5, 2023

Conversation

unknownbrackets
Copy link
Collaborator

This adds a cheap priority mechanism to mark certain tasks as higher priority, as a basic way to manage dependencies.

Of course, if a HIGH is enqueued while a LOW is executing, it won't change anything. But as long as HIGH/NORMAL/LOW are enqueued near the same time, this ensures that the HIGHs would be scheduled first. Before, that wasn't guaranteed (even if you enqueued the tasks in the right order), because things would go onto the global queue and get picked up, which would also vary by thread count.

-[Unknown]

@unknownbrackets unknownbrackets added this to the v1.15.0 milestone Jan 15, 2023
@hrydgard
Copy link
Owner

This seems fine, but for some reason doesn't solve the problem. Will look more later, for now here are the stacks when force-quitting in the hanged state:

 Type:        EXC_CRASH (SIGQUIT)
Exception Codes:       0x0000000000000000, 0x0000000000000000

Termination Reason:    Namespace SIGNAL, Code 3 Quit: 3
Terminating Process:   Terminal [1383]

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib        	       0x193041564 __psynch_cvwait + 8
1   libsystem_pthread.dylib       	       0x19307d638 _pthread_cond_wait + 1232
2   libc++.1.dylib                	       0x192fcaac4 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 28
3   PPSSPPSDL                     	       0x1044ceb6c VulkanRenderManager::BeginFrame(bool, bool) + 136
4   PPSSPPSDL                     	       0x1044c72c4 Draw::VKContext::BeginFrame() + 40
5   PPSSPPSDL                     	       0x1040f0458 EmuScreen::preRender() + 44
6   PPSSPPSDL                     	       0x1044f77e4 ScreenManager::render() + 200
7   PPSSPPSDL                     	       0x1040c8e38 NativeRender(GraphicsContext*) + 596
8   PPSSPPSDL                     	       0x10451d9f0 main + 5368
9   dyld                          	       0x192d53e50 start + 2544

Thread 1:
0   libsystem_pthread.dylib       	       0x193077e18 start_wqthread + 0

Thread 2:
0   libsystem_pthread.dylib       	       0x193077e18 start_wqthread + 0

Thread 3:
0   libsystem_pthread.dylib       	       0x193077e18 start_wqthread + 0

Thread 4:: caulk.messenger.shared:17
0   libsystem_kernel.dylib        	       0x19303dcec semaphore_wait_trap + 8
1   caulk                         	       0x19c556cfc caulk::mach::semaphore::wait_or_error() + 28
2   caulk                         	       0x19c539634 caulk::concurrent::details::worker_thread::run() + 56
3   caulk                         	       0x19c539278 void* caulk::thread_proxy<std::__1::tuple<caulk::thread::attributes, void (caulk::concurrent::details::worker_thread::*)(), std::__1::tuple<caulk::concurrent::details::worker_thread*> > >(void*) + 96
4   libsystem_pthread.dylib       	       0x19307d06c _pthread_start + 148
5   libsystem_pthread.dylib       	       0x193077e2c thread_start + 8

Thread 5:: PoolWorker 0
0   libsystem_kernel.dylib        	       0x193041564 __psynch_cvwait + 8
1   libsystem_pthread.dylib       	       0x19307d638 _pthread_cond_wait + 1232
2   libc++.1.dylib                	       0x192fcaac4 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 28
3   PPSSPPSDL                     	       0x1043b2e7c Promise<VkShaderModule_T*>::BlockUntilReady() + 100
4   PPSSPPSDL                     	       0x1044cc1d0 VKRGraphicsPipeline::Create(VulkanContext*, VkRenderPass_T*, RenderPassType, VkSampleCountFlagBits, double) + 180
5   PPSSPPSDL                     	       0x1044d2008 CreateMultiPipelinesTask::Run() + 52
6   PPSSPPSDL                     	       0x1044f2010 WorkerThreadFunc(GlobalThreadContext*, ThreadContext*) + 312
7   PPSSPPSDL                     	       0x1044f3168 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(GlobalThreadContext*, ThreadContext*), GlobalThreadContext*, ThreadContext*> >(void*) + 48
8   libsystem_pthread.dylib       	       0x19307d06c _pthread_start + 148
9   libsystem_pthread.dylib       	       0x193077e2c thread_start + 8

Thread 6:: PoolWorker 1
0   libsystem_kernel.dylib        	       0x193041564 __psynch_cvwait + 8
1   libsystem_pthread.dylib       	       0x19307d638 _pthread_cond_wait + 1232
2   libc++.1.dylib                	       0x192fcaac4 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 28
3   PPSSPPSDL                     	       0x1043b2e7c Promise<VkShaderModule_T*>::BlockUntilReady() + 100
4   PPSSPPSDL                     	       0x1044cc1d0 VKRGraphicsPipeline::Create(VulkanContext*, VkRenderPass_T*, RenderPassType, VkSampleCountFlagBits, double) + 180
5   PPSSPPSDL                     	       0x1044d2008 CreateMultiPipelinesTask::Run() + 52
6   PPSSPPSDL                     	       0x1044f2010 WorkerThreadFunc(GlobalThreadContext*, ThreadContext*) + 312
7   PPSSPPSDL                     	       0x1044f3168 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(GlobalThreadContext*, ThreadContext*), GlobalThreadContext*, ThreadContext*> >(void*) + 48
8   libsystem_pthread.dylib       	       0x19307d06c _pthread_start + 148
9   libsystem_pthread.dylib       	       0x193077e2c thread_start + 8

Thread 7:: PoolWorkerIO 2
0   libsystem_kernel.dylib        	       0x193041564 __psynch_cvwait + 8
1   libsystem_pthread.dylib       	       0x19307d638 _pthread_cond_wait + 1232
2   libc++.1.dylib                	       0x192fcaac4 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 28
3   PPSSPPSDL                     	       0x1044f222c WorkerThreadFunc(GlobalThreadContext*, ThreadContext*) + 852
4   PPSSPPSDL                     	       0x1044f3168 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(GlobalThreadContext*, ThreadContext*), GlobalThreadContext*, ThreadContext*> >(void*) + 48
5   libsystem_pthread.dylib       	       0x19307d06c _pthread_start + 148
6   libsystem_pthread.dylib       	       0x193077e2c thread_start + 8

Thread 8:: PoolWorkerIO 3
0   libsystem_kernel.dylib        	       0x193041564 __psynch_cvwait + 8
1   libsystem_pthread.dylib       	       0x19307d638 _pthread_cond_wait + 1232
2   libc++.1.dylib                	       0x192fcaac4 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 28
3   PPSSPPSDL                     	       0x1044f222c WorkerThreadFunc(GlobalThreadContext*, ThreadContext*) + 852
4   PPSSPPSDL                     	       0x1044f3168 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(GlobalThreadContext*, ThreadContext*), GlobalThreadContext*, ThreadContext*> >(void*) + 48
5   libsystem_pthread.dylib       	       0x19307d06c _pthread_start + 148
6   libsystem_pthread.dylib       	       0x193077e2c thread_start + 8

Thread 9:: PoolWorkerIO 4
0   libsystem_kernel.dylib        	       0x193041564 __psynch_cvwait + 8
1   libsystem_pthread.dylib       	       0x19307d638 _pthread_cond_wait + 1232
2   libc++.1.dylib                	       0x192fcaac4 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 28
3   PPSSPPSDL                     	       0x1044f222c WorkerThreadFunc(GlobalThreadContext*, ThreadContext*) + 852
4   PPSSPPSDL                     	       0x1044f3168 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(GlobalThreadContext*, ThreadContext*), GlobalThreadContext*, ThreadContext*> >(void*) + 48
5   libsystem_pthread.dylib       	       0x19307d06c _pthread_start + 148
6   libsystem_pthread.dylib       	       0x193077e2c thread_start + 8

Thread 10:: PoolWorkerIO 5
0   libsystem_kernel.dylib        	       0x193041564 __psynch_cvwait + 8
1   libsystem_pthread.dylib       	       0x19307d638 _pthread_cond_wait + 1232
2   libc++.1.dylib                	       0x192fcaac4 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 28
3   PPSSPPSDL                     	       0x1044f222c WorkerThreadFunc(GlobalThreadContext*, ThreadContext*) + 852
4   PPSSPPSDL                     	       0x1044f3168 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(GlobalThreadContext*, ThreadContext*), GlobalThreadContext*, ThreadContext*> >(void*) + 48
5   libsystem_pthread.dylib       	       0x19307d06c _pthread_start + 148
6   libsystem_pthread.dylib       	       0x193077e2c thread_start + 8

Thread 11:: AudioQueue thread
0   libsystem_kernel.dylib        	       0x19303dd70 mach_msg2_trap + 8
1   libsystem_kernel.dylib        	       0x19304f8a4 mach_msg2_internal + 80
2   libsystem_kernel.dylib        	       0x1930465c4 mach_msg_overwrite + 540
3   libsystem_kernel.dylib        	       0x19303e0ec mach_msg + 24
4   CoreFoundation                	       0x19315cbc0 __CFRunLoopServiceMachPort + 160
5   CoreFoundation                	       0x19315b4ac __CFRunLoopRun + 1232
6   CoreFoundation                	       0x19315a888 CFRunLoopRunSpecific + 612
7   libSDL2-2.0.0.dylib           	       0x105912aac audioqueue_thread + 1152
8   libSDL2-2.0.0.dylib           	       0x1058aa18c SDL_RunThread + 48
9   libSDL2-2.0.0.dylib           	       0x1058ff6cc RunThread + 12
10  libsystem_pthread.dylib       	       0x19307d06c _pthread_start + 148
11  libsystem_pthread.dylib       	       0x193077e2c thread_start + 8

Thread 12:: caulk.messenger.shared:high
0   libsystem_kernel.dylib        	       0x19303dcec semaphore_wait_trap + 8
1   caulk                         	       0x19c556cfc caulk::mach::semaphore::wait_or_error() + 28
2   caulk                         	       0x19c539634 caulk::concurrent::details::worker_thread::run() + 56
3   caulk                         	       0x19c539278 void* caulk::thread_proxy<std::__1::tuple<caulk::thread::attributes, void (caulk::concurrent::details::worker_thread::*)(), std::__1::tuple<caulk::concurrent::details::worker_thread*> > >(void*) + 96
4   libsystem_pthread.dylib       	       0x19307d06c _pthread_start + 148
5   libsystem_pthread.dylib       	       0x193077e2c thread_start + 8

Thread 13:: AQConverterThread
0   libsystem_kernel.dylib        	       0x193041564 __psynch_cvwait + 8
1   libsystem_pthread.dylib       	       0x19307d638 _pthread_cond_wait + 1232
2   libAudioToolboxUtility.dylib  	       0x1a025f1e0 CADeprecated::CAGuard::Wait() + 60
3   AudioToolbox                  	       0x1a146abbc AQConverterManager::AQConverterThread::ConverterThreadEntry(void*) + 720
4   libAudioToolboxUtility.dylib  	       0x1a02405c0 CADeprecated::CAPThread::Entry(CADeprecated::CAPThread*) + 92
5   libsystem_pthread.dylib       	       0x19307d06c _pthread_start + 148
6   libsystem_pthread.dylib       	       0x193077e2c thread_start + 8

Thread 14:: com.apple.audio.IOThread.client
0   libsystem_kernel.dylib        	       0x19303dd70 mach_msg2_trap + 8
1   libsystem_kernel.dylib        	       0x19304f8a4 mach_msg2_internal + 80
2   libsystem_kernel.dylib        	       0x1930465c4 mach_msg_overwrite + 540
3   libsystem_kernel.dylib        	       0x19303e0ec mach_msg + 24
4   CoreAudio                     	       0x195489724 HALB_MachPort::SendSimpleMessageWithSimpleReply(unsigned int, unsigned int, int, int&, bool, unsigned int) + 104
5   CoreAudio                     	       0x19535bff0 HALC_ProxyIOContext::IOWorkLoop() + 3612
6   CoreAudio                     	       0x19535ab8c invocation function for block in HALC_ProxyIOContext::HALC_ProxyIOContext(unsigned int, unsigned int) + 116
7   CoreAudio                     	       0x1954e5654 HALB_IOThread::Entry(void*) + 88
8   libsystem_pthread.dylib       	       0x19307d06c _pthread_start + 148
9   libsystem_pthread.dylib       	       0x193077e2c thread_start + 8

Thread 15:
0   libsystem_kernel.dylib        	       0x193041564 __psynch_cvwait + 8
1   libsystem_pthread.dylib       	       0x19307d638 _pthread_cond_wait + 1232
2   libc++.1.dylib                	       0x192fcab60 std::__1::condition_variable::__do_timed_wait(std::__1::unique_lock<std::__1::mutex>&, std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >) + 100
3   PPSSPPSDL                     	       0x104c6f95c IoThreadHolder::Start()::'lambda'()::operator()() const + 276
4   PPSSPPSDL                     	       0x104c6f7e8 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, IoThreadHolder::Start()::'lambda'()> >(void*) + 44
5   libsystem_pthread.dylib       	       0x19307d06c _pthread_start + 148
6   libsystem_pthread.dylib       	       0x193077e2c thread_start + 8

Thread 16:: RenderMan
0   libsystem_kernel.dylib        	       0x193041564 __psynch_cvwait + 8
1   libsystem_pthread.dylib       	       0x19307d638 _pthread_cond_wait + 1232
2   libc++.1.dylib                	       0x192fcaac4 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 28
3   PPSSPPSDL                     	       0x1044cc824 Promise<VkPipeline_T*>::BlockUntilReady() + 100
4   PPSSPPSDL                     	       0x1044d70cc VulkanQueueRunner::PerformRenderPass(VKRStep const&, VkCommandBuffer_T*) + 2752
5   PPSSPPSDL                     	       0x1044d6284 VulkanQueueRunner::RunSteps(std::__1::vector<VKRStep*, std::__1::allocator<VKRStep*> >&, FrameData&, FrameDataShared&, bool) + 436
6   PPSSPPSDL                     	       0x1044ce9cc VulkanRenderManager::Run(VKRRenderThreadTask&) + 408
7   PPSSPPSDL                     	       0x1044cd6c0 VulkanRenderManager::ThreadFunc() + 312
8   PPSSPPSDL                     	       0x1044d26ac void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (VulkanRenderManager::*)(), VulkanRenderManager*> >(void*) + 64
9   libsystem_pthread.dylib       	       0x19307d06c _pthread_start + 148
10  libsystem_pthread.dylib       	       0x193077e2c thread_start + 8

Thread 17:: ShaderCompile
0   libsystem_kernel.dylib        	       0x193041564 __psynch_cvwait + 8
1   libsystem_pthread.dylib       	       0x19307d638 _pthread_cond_wait + 1232
2   libc++.1.dylib                	       0x192fcaac4 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 28
3   PPSSPPSDL                     	       0x1044cd824 VulkanRenderManager::CompileThreadFunc() + 192
4   PPSSPPSDL                     	       0x1044d26ac void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (VulkanRenderManager::*)(), VulkanRenderManager*> >(void*) + 64
5   libsystem_pthread.dylib       	       0x19307d06c _pthread_start + 148
6   libsystem_pthread.dylib       	       0x193077e2c thread_start + 8

Thread 18:: com.apple.NSEventThread
0   libsystem_kernel.dylib        	       0x19303dd70 mach_msg2_trap + 8
1   libsystem_kernel.dylib        	       0x19304f8a4 mach_msg2_internal + 80
2   libsystem_kernel.dylib        	       0x1930465c4 mach_msg_overwrite + 540
3   libsystem_kernel.dylib        	       0x19303e0ec mach_msg + 24
4   CoreFoundation                	       0x19315cbc0 __CFRunLoopServiceMachPort + 160
5   CoreFoundation                	       0x19315b4ac __CFRunLoopRun + 1232
6   CoreFoundation                	       0x19315a888 CFRunLoopRunSpecific + 612
7   AppKit                        	       0x196506410 _NSEventThread + 172
8   libsystem_pthread.dylib       	       0x19307d06c _pthread_start + 148
9   libsystem_pthread.dylib       	       0x193077e2c thread_start + 8

Thread 19:: IO
0   libsystem_kernel.dylib        	       0x193041564 __psynch_cvwait + 8
1   libsystem_pthread.dylib       	       0x19307d638 _pthread_cond_wait + 1232
2   libc++.1.dylib                	       0x192fcaac4 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 28
3   PPSSPPSDL                     	       0x10428678c void std::__1::condition_variable_any::wait<std::__1::unique_lock<std::__1::recursive_mutex> >(std::__1::unique_lock<std::__1::recursive_mutex>&) + 100
4   PPSSPPSDL                     	       0x104286348 ThreadEventQueue<NoBase, AsyncIOEvent, AsyncIOEventType, (AsyncIOEventType)0, (AsyncIOEventType)1, (AsyncIOEventType)2>::RunEventsUntil(unsigned long long) + 92
5   PPSSPPSDL                     	       0x104284150 __IoManagerThread() + 164
6   PPSSPPSDL                     	       0x1041e5f10 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)()> >(void*) + 44
7   libsystem_pthread.dylib       	       0x19307d06c _pthread_start + 148
8   libsystem_pthread.dylib       	       0x193077e2c thread_start + 8

Thread 20:: SAS
0   libsystem_kernel.dylib        	       0x193041564 __psynch_cvwait + 8
1   libsystem_pthread.dylib       	       0x19307d638 _pthread_cond_wait + 1232
2   libc++.1.dylib                	       0x192fcaac4 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 28
3   PPSSPPSDL                     	       0x10431c674 __SasThread() + 148
4   PPSSPPSDL                     	       0x10431f8c0 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()> >(void*) + 44
5   libsystem_pthread.dylib       	       0x19307d06c _pthread_start + 148
6   libsystem_pthread.dylib       	       0x193077e2c thread_start + 8

Thread 21:: UPnPService
0   libsystem_kernel.dylib        	       0x193041360 __semwait_signal + 8
1   libsystem_c.dylib             	       0x192f4b5b0 nanosleep + 220
2   libsystem_c.dylib             	       0x192f4b4c8 usleep + 68
3   PPSSPPSDL                     	       0x104388584 upnpService(unsigned int) + 160
4   PPSSPPSDL                     	       0x10438ab60 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)(unsigned int), unsigned int> >(void*) + 48
5   libsystem_pthread.dylib       	       0x19307d06c _pthread_start + 148
6   libsystem_pthread.dylib       	       0x193077e2c thread_start + 8

@unknownbrackets
Copy link
Collaborator Author

Hm, it does only have 2 CPU threads but it should work. We might need to do some task logging like this:


			static std::atomic<int> n;
			int id = n++;
			INFO_LOG(COMMON, "%s running task priority=%d, id=%d", task->Priority(), id);
			INFO_LOG(COMMON, "Queued status:");
			for (size_t p = 0; p < TASK_PRIORITY_COUNT; ++p) {
				std::unique_lock<std::mutex> lock(thread->mutex);
				for (auto &t : thread->private_queue[p])
					INFO_LOG(COMMON, "Thread (%s): priority=%d", t->Priority());
			}
			for (size_t p = 0; p < TASK_PRIORITY_COUNT; ++p) {
				std::unique_lock<std::mutex> lock(global->mutex);
				for (auto &t : global->compute_queue[p])
					INFO_LOG(COMMON, "Global: compute, priority=%d", t->Priority());
				for (auto &t : global->io_queue[p])
					INFO_LOG(COMMON, "Global: io, priority=%d", t->Priority());
			}
			task->Run();
			task->Release();
			INFO_LOG(COMMON, "Finished %d", id);

Would be useful to see what task it last pulls from the queue and what else is on the queue at the time.

-[Unknown]

@hrydgard
Copy link
Owner

hrydgard commented Feb 1, 2023

I resolved my dependency/deadlock problem in a different way in #16802, but I think I still want this in. One use case is to prioritize pipeline creation higher than texture replacement loading, for example.

There's some tiny conflict, could you resolve? I'll review in detail then, but at a cursory look this looks good.

@unknownbrackets
Copy link
Collaborator Author

Updated and set the priority to high for the multi pipeline task.

-[Unknown]

@hrydgard
Copy link
Owner

hrydgard commented Feb 4, 2023

Cool. Sorry for the delay, I'll review and most likely merge tomorrow.

Copy link
Owner

@hrydgard hrydgard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the name of the PR I was expecting the data structure known as priority queue, which would have been a bit of overkill for this :) This simpler approach is more appropriate and good enough for the intended use.

@hrydgard hrydgard merged commit ec5afb2 into hrydgard:master Feb 5, 2023
@unknownbrackets unknownbrackets deleted the task-priority branch February 5, 2023 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants