User Panel
User Panel
User Panel
USA
|
Posted: 2/28/2024 5:42:04 AM EDT
I see a lot of new laptops being marketed as something to the effective as being designed for AI, even some very low-power laptops with integrated graphics.
What do they mean by local AI? I thought using local LLMs was very compute intensive, and typically needed GPU or other suitable processors (ie not something that a 1.2lb laptop is going to support). |
|
|
|
Posted: 2/28/2024 7:53:35 AM EDT
[#1]
Not sure about others but Apple's mobile processors have a Neural engine for AI
https://www.macobserver.com/tips/deep-dive/what-is-apple-neural-engine/ |
|
|
|
|
Posted: 2/28/2024 8:28:42 AM EDT
[#2]
At CES they had monitors that contained AI
The future is now 
|
|
|
|
USA
|
Posted: 2/28/2024 3:21:43 PM EDT
[#3]
Quote HistoryOriginally Posted By Tejas1836: Not sure about others but Apple's mobile processors have a Neural engine for AI https://www.macobserver.com/tips/deep-dive/what-is-apple-neural-engine/ Interesting... So it sounds like it's not designed to support an entire LLM on the machine, but rather to support various features of different software (for instance, Siri was mentioned in the article)? |
|
|
|
Posted: 3/1/2024 10:20:27 PM EDT
[#4]
You can run local LLMs on the M series Macs if you have enough memory.
LM Studio, llamafile, mlx are all tools that allow working with them. |
|
|
|
|
Posted: 3/2/2024 11:29:02 AM EDT
[#5]
I'm running local LLMs on PCs. You could probably run a smaller Q on a laptop but to run the larger Q you benefit from a high end GPU. The speed is just too slow IMO without a good GPU and a GPU offload.
|
|
|
|
|
Posted: 3/2/2024 11:36:06 AM EDT
[#6]
Quote HistoryOriginally Posted By Josh: You can run local LLMs on the M series Macs if you have enough memory. LM Studio, llamafile, mlx are all tools that allow working with them. Memory driven execution is a lot slower in my experience. It works though. I'd add PrivateGPT as another to research. The install is sort of a PITA due to all but one video on YT being inaccurate after a change by the creator. If any of y'all want it just @ me here and I will find the link to the correct method. I think only windows users had an issue with the big update. |
|
|
|
|
Posted: 3/2/2024 11:45:29 AM EDT
[#7]
Quote HistoryOriginally Posted By JTX23: Memory driven execution is a lot slower in my experience. It works though. I'd add PrivateGPT as another to research. The install is sort of a PITA due to all but one video on YT being inaccurate after a change by the creator. If any of y'all want it just @ me here and I will find the link to the correct method. I think only windows users had an issue with the big update. Quote HistoryOriginally Posted By JTX23: Originally Posted By Josh: You can run local LLMs on the M series Macs if you have enough memory. LM Studio, llamafile, mlx are all tools that allow working with them. Memory driven execution is a lot slower in my experience. It works though. I'd add PrivateGPT as another to research. The install is sort of a PITA due to all but one video on YT being inaccurate after a change by the creator. If any of y'all want it just @ me here and I will find the link to the correct method. I think only windows users had an issue with the big update. Macs have GPU & shared GPU memory. Great performance on the M1-M3 MacBook Pros. PrivateGPT is a suite of tools to run LLMs using local documents as a data source. |
|
|
|
|
Posted: 3/2/2024 11:46:31 AM EDT
[#8]
Originally Posted By Flockas: I thought using local LLMs was very compute intensive, and typically needed GPU or other suitable processors (ie not something that a 1.2lb laptop is going to support). It is a resource hog but you can run it at different quantizations. Each file will have a Q followed by a number like Q3. The larger the number the better hardware you are going to need. The performance of the model goes up with the number. Depending on what you want Q3 is not bad. I'm an engineer with a lot of ideas but not the programming skills to see them all through on my own. I'm leveraging LLMs to help. I use Q3 for rapid response code snippets but Q5 for more complex stuff. More of the brainstorming and planning. From what I can tell if you want to ingest documents you need vram and a good amount. That is the stepping point where I think there is only the PC path. |
|
|
|
|
Posted: 3/2/2024 11:48:26 AM EDT
[#9]
Quote HistoryOriginally Posted By Josh: Macs have GPU & shared GPU memory. Great performance on the M1-M3 MacBook Pros. PrivateGPT is a suite of tools to run LLMs using local documents as a data source. Good deal. I'm not familiar with macs. Out of curiosity how much vram does a mid range macbook pro come with? |
|
|
|
|
Posted: 3/2/2024 2:12:25 PM EDT
[Last Edit: Josh]
[#10]
Quote HistoryOriginally Posted By JTX23: Good deal. I'm not familiar with macs. Out of curiosity how much vram does a mid range macbook pro come with? Quote HistoryOriginally Posted By JTX23: Originally Posted By Josh: Macs have GPU & shared GPU memory. Great performance on the M1-M3 MacBook Pros. PrivateGPT is a suite of tools to run LLMs using local documents as a data source. Good deal. I'm not familiar with macs. Out of curiosity how much vram does a mid range macbook pro come with? All memory is shared memory between the GPU and the CPU in the M series Macs now. It's all on the same die. You can get them anywhere from 8GB to 128GB. I can run Mixtral 8x7B-Q5 just fine on my M1 64GB. https://www.macrumors.com/guide/m1/ |
|
|
|
|
Posted: 3/3/2024 12:40:35 AM EDT
[#11]
Quote HistoryOriginally Posted By JTX23: Good deal. I'm not familiar with macs. Out of curiosity how much vram does a mid range macbook pro come with? Quote HistoryOriginally Posted By JTX23: Originally Posted By Josh: Macs have GPU & shared GPU memory. Great performance on the M1-M3 MacBook Pros. PrivateGPT is a suite of tools to run LLMs using local documents as a data source. Good deal. I'm not familiar with macs. Out of curiosity how much vram does a mid range macbook pro come with? FYI I got privategpt running, using mistral 7b-instruct-q4. Runs like a champ on my Mac. Adequately summarizes documents and answers questions based on them. Only minor difficulty in installation was getting the right python version installed, activated, and then getting poetry installed and running. Other than that everything worked the way it was supposed to. Their OSX instructions on getting pyenv installed don't work exactly as they explain them. |
|
|
|
|
Posted: 3/3/2024 2:39:04 AM EDT
[#12]
Can you get dolphin mixtral to run on privateGPT?
I changed a few lines in the settings file to point it to the model. It loads but hangs up on questions. I'm sure I'm doing something wrong but haven't had the time to dig into it yet. |
|
|
|
|
Posted: 3/3/2024 9:03:11 AM EDT
[#13]
Quote HistoryOriginally Posted By JTX23: Can you get dolphin mixtral to run on privateGPT? I changed a few lines in the settings file to point it to the model. It loads but hangs up on questions. I'm sure I'm doing something wrong but haven't had the time to dig into it yet. I’m sure you can but I haven’t tried. |
|
|
|
|
Posted: 3/3/2024 6:31:25 PM EDT
[#14]
Quote HistoryOriginally Posted By Josh: All memory is shared memory between the GPU and the CPU in the M series Macs now. It's all on the same die. You can get them anywhere from 8GB to 128GB. I can run Mixtral 8x7B-Q5 just fine on my M1 64GB. https://www.macrumors.com/guide/m1/ Quote HistoryOriginally Posted By Josh: Originally Posted By JTX23: Originally Posted By Josh: Macs have GPU & shared GPU memory. Great performance on the M1-M3 MacBook Pros. PrivateGPT is a suite of tools to run LLMs using local documents as a data source. Good deal. I'm not familiar with macs. Out of curiosity how much vram does a mid range macbook pro come with? All memory is shared memory between the GPU and the CPU in the M series Macs now. It's all on the same die. You can get them anywhere from 8GB to 128GB. I can run Mixtral 8x7B-Q5 just fine on my M1 64GB. https://www.macrumors.com/guide/m1/  put another way, the Apple silicon SOCs were designed from day 1 to run AI on machine. Being able to use any of the system resources on demand turns out has a better impact on performance then a dedicated (but larger) separate pool on memory. MacOS also dynamically schedules processes so if the GPU is busy it can determine if it's faster to wait on GPU or run on CPU. or Vice versa. It can also schedule tasks to run on the neural processor if the task is optimized for it. |
|
|
|
|
Posted: 3/3/2024 7:34:29 PM EDT
[#15]
Quote HistoryOriginally Posted By right_rudder: For the record Josh's M1 may or may not have been the test bed for a yet to be released "local" AI. put another way, the Apple silicon SOCs were designed from day 1 to run AI on machine. Being able to use any of the system resources on demand turns out has a better impact on performance then a dedicated (but larger) separate pool on memory. MacOS also dynamically schedules processes so if the GPU is busy it can determine if it's faster to wait on GPU or run on CPU. or Vice versa. It can also schedule tasks to run on the neural processor if the task is optimized for it. Quote HistoryOriginally Posted By right_rudder: Originally Posted By Josh: Originally Posted By JTX23: Originally Posted By Josh: Macs have GPU & shared GPU memory. Great performance on the M1-M3 MacBook Pros. PrivateGPT is a suite of tools to run LLMs using local documents as a data source. Good deal. I'm not familiar with macs. Out of curiosity how much vram does a mid range macbook pro come with? All memory is shared memory between the GPU and the CPU in the M series Macs now. It's all on the same die. You can get them anywhere from 8GB to 128GB. I can run Mixtral 8x7B-Q5 just fine on my M1 64GB. https://www.macrumors.com/guide/m1/ put another way, the Apple silicon SOCs were designed from day 1 to run AI on machine. Being able to use any of the system resources on demand turns out has a better impact on performance then a dedicated (but larger) separate pool on memory. MacOS also dynamically schedules processes so if the GPU is busy it can determine if it's faster to wait on GPU or run on CPU. or Vice versa. It can also schedule tasks to run on the neural processor if the task is optimized for it.  I am absolutely astonished at how much of an improvement this machine is over the i9 I've been using for the last several years. I had one from work up to about a year ago when I switched jobs -- I was not happy to give it up. I'm trying to get privateGPT at my job, it would make it tremendously easier dealing with what I have to do there. |
|
|
|
|
Posted: 3/3/2024 8:47:18 PM EDT
[#16]
Quote HistoryOriginally Posted By Josh: I am absolutely astonished at how much of an improvement this machine is over the i9 I've been using for the last several years. I had one from work up to about a year ago when I switched jobs -- I was not happy to give it up. I'm trying to get privateGPT at my job, it would make it tremendously easier dealing with what I have to do there. it was literally just sitting on my other desk mining xmr. |
|
|
|
|
Posted: 3/3/2024 10:12:47 PM EDT
[#17]
Quote HistoryOriginally Posted By right_rudder: I'm glad you ended up with it and it went to a good home where it'll get good use. it was literally just sitting on my other desk mining xmr. I appreciate the deal you made me on it. I'll have to try the xmr thing sometime. |
|
|
|
Win a FREE Membership!
Sign up for the ARFCOM weekly newsletter and be entered to win a free ARFCOM membership. One new winner* is announced every week!
You will receive an email every Friday morning featuring the latest chatter from the hottest topics, breaking news surrounding legislation, as well as exclusive deals only available to ARFCOM email subscribers.
AR15.COM is the world's largest firearm community and is a gathering place for firearm enthusiasts of all types.
From hunters and military members, to competition shooters and general firearm enthusiasts, we welcome anyone who values and respects the way of the firearm.
Subscribe to our monthly Newsletter to receive firearm news, product discounts from your favorite Industry Partners, and more.
Copyright © 1996-2024 AR15.COM LLC. All Rights Reserved.
Any use of this content without express written consent is prohibited.
AR15.Com reserves the right to overwrite or replace any affiliate, commercial, or monetizable links, posted by users, with our own.
