０７６萌ＳＮＳ
  • Login

  • Public

    • Public
    • Network
    • Groups
    • Popular
    • People

Notices by rees (rees@breastmilk.club), page 13

  1. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 23:35:29 JST rees rees
    in reply to
    • Moon
    • Machismo
    • NEETzsche
    • feld
    • FrailLeaf
    • Fantasia
    @NEETzsche @feld @fantasia @Zerglingman @FrailLeaf @Moon https://arxiv.org/abs/2310.01889
    https://arxiv.org/abs/2310.08560
    https://syncedreview.com/2021/09/09/deepmind-podracer-tpu-based-rl-frameworks-deliver-exceptional-performance-at-low-cost-100/
    In conversation about a year ago from social.076.moe permalink

    Attachments

    1. Ring Attention with Blockwise Transformers for Near-Infinite Context
      Transformers have emerged as the architecture of choice for many state-of-the-art AI models, showcasing exceptional performance across a wide range of AI applications. However, the memory demands imposed by Transformers limit their ability to handle long sequences, thereby posing challenges in utilizing videos, actions, and other long-form sequences and modalities in complex environments. We present a novel approach, Ring Attention with Blockwise Transformers (Ring Attention), which leverages blockwise computation of self-attention and feedforward to distribute long sequences across multiple devices while fully overlapping the communication of key-value blocks with the computation of blockwise attention. Our approach enables training and inference of sequences that are up to device count times longer than those achievable by prior memory-efficient Transformers, without resorting to approximations or incurring additional communication and computation overheads. Extensive experiments on language modeling and reinforcement learning tasks demonstrate the effectiveness of our approach in allowing millions of tokens context size and improving performance.
    2. MemGPT: Towards LLMs as Operating Systems
      Large language models (LLMs) have revolutionized AI, but are constrained by limited context windows, hindering their utility in tasks like extended conversations and document analysis. To enable using context beyond limited context windows, we propose virtual context management, a technique drawing inspiration from hierarchical memory systems in traditional operating systems that provide the appearance of large memory resources through data movement between fast and slow memory. Using this technique, we introduce MemGPT (Memory-GPT), a system that intelligently manages different memory tiers in order to effectively provide extended context within the LLM's limited context window, and utilizes interrupts to manage control flow between itself and the user. We evaluate our OS-inspired design in two domains where the limited context windows of modern LLMs severely handicaps their performance: document analysis, where MemGPT is able to analyze large documents that far exceed the underlying LLM's context window, and multi-session chat, where MemGPT can create conversational agents that remember, reflect, and evolve dynamically through long-term interactions with their users. We release MemGPT code and data for our experiments at https://memgpt.ai.
    3. Infinite Memory Transformer: Attending to Arbitrarily Long Contexts Without Increasing Computation Burden
      from Synced
      When reading a novel, humans naturally remember relevant plot information even if it was presented many chapters earlier. Although today’s transformer-based language models have made impressive progress in natural language processing, they struggle in this regard, as the compute required for modelling long-term memories grows quadratically with the length of the text and will eventually
  2. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 23:34:30 JST rees rees
    in reply to
    • Moon
    • Machismo
    • NEETzsche
    • feld
    • teknomunk
    @teknomunk @feld @Zerglingman @NEETzsche @Moon I have been programming since I was 8, sounds like you are coping
    In conversation about a year ago from breastmilk.club permalink
  3. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 23:34:00 JST rees rees
    in reply to
    • Moon
    • Machismo
    • NEETzsche
    • feld
    • FrailLeaf
    • Fantasia
    @FrailLeaf @feld @fantasia @Zerglingman @NEETzsche @Moon infinite context windows are already a thing
    In conversation about a year ago from breastmilk.club permalink
  4. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 23:33:16 JST rees rees
    in reply to
    • Moon
    • Machismo
    • NEETzsche
    • feld
    • Fantasia
    @NEETzsche @feld @fantasia @Zerglingman @Moon i'm fully convinced
    In conversation about a year ago from social.076.moe permalink
  5. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 23:32:50 JST rees rees
    in reply to
    • Moon
    • Machismo
    • NEETzsche
    • feld
    • Fantasia
    @fantasia @feld @Zerglingman @NEETzsche @Moon lol ok so wait literally two years
    In conversation about a year ago from social.076.moe permalink
  6. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 23:29:45 JST rees rees
    in reply to
    • Moon
    • Machismo
    • NEETzsche
    • feld
    • Fantasia
    @fantasia @feld @Zerglingman @NEETzsche @Moon LLMs already can correct code/logic errors, what you mean to say is "until they can be more consistent"
    In conversation about a year ago from social.076.moe permalink

    Attachments


    1. https://maymays.breastmilk.club/media/eed721638d9442aa9360e9de04339a6820dcd8d3f9e83d957ec212d71b1d2757.png
  7. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 23:27:16 JST rees rees
    in reply to
    • Moon
    • Machismo
    • NEETzsche
    • feld
    @Zerglingman @feld @NEETzsche @Moon NO BULLY :(
    In conversation about a year ago from social.076.moe permalink
  8. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 23:26:48 JST rees rees
    in reply to
    • Moon
    • Machismo
    • NEETzsche
    • feld
    @NEETzsche @feld @Zerglingman @Moon LLM + some sort of hypervisor agent that coordinates them
    In conversation about a year ago from social.076.moe permalink
  9. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 23:25:50 JST rees rees
    in reply to
    • Moon
    • Machismo
    • NEETzsche
    • feld
    @feld @Zerglingman @NEETzsche @Moon gonna get heat for this but programming is on its way out anyways so who cares if it's elixir or w/e
    In conversation about a year ago from social.076.moe permalink
  10. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 23:23:54 JST rees rees
    in reply to
    • Moon
    • Machismo
    • NEETzsche
    @NEETzsche @Zerglingman @Moon that's one reason, I actually find it quicker to prototype in rust than javascript at times
    In conversation about a year ago from breastmilk.club permalink
  11. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 23:20:08 JST rees rees
    in reply to
    • Moon
    • Machismo
    • NEETzsche
    @NEETzsche @Zerglingman @Moon >During this process, we were able to remove over 400 unsafe expressions while discovering and fixing several hard-to-spot concurrency bugs along the way.
    In conversation about a year ago from breastmilk.club permalink
  12. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 23:19:18 JST rees rees
    in reply to
    • Moon
    • Machismo
    • NEETzsche
    @NEETzsche @Zerglingman @Moon I linked you to a study (and pete, got no replies) on rust creating sound code for kernels

    https://dl.acm.org/doi/10.1145/3623759.3624554
    >The memory-safe systems programming language Rust is gaining more and more attention in the operating system development communities, as it provides memory safety without sacrificing performance or control. However, these safety guarantees only apply to the safe subset of Rust, while bare-metal programming requires some parts of the program to be written in unsafe Rust. Writing abstractions for these parts of the software that are sound, meaning that they guarantee the absence of undefined behavior and thus uphold the invariants of safe Rust, can be challenging. Producing sound code, however, is essential to avoid breakage when the code is used in new ways or the compiler behavior changes.
    >In this paper, we present common patterns of unsound abstractions derived from the experience of reworking soundness in our kernel. During this process, we were able to remove over 400 unsafe expressions while discovering and fixing several hard-to-spot concurrency bugs along the way.
    In conversation about a year ago from breastmilk.club permalink

    Attachments

    1. On the Challenge of Sound Code for Operating Systems | Proceedings of the 12th Workshop on Programming Languages and Operating Systems
  13. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 23:08:58 JST rees rees
    in reply to
    • Moon
    • NEETzsche
    @NEETzsche @Moon >is a meme language
    use real words, no one knows what you mean by "meme language" at this point when JS is the majority of the web and has more usage than your non-meme languages
    In conversation about a year ago from breastmilk.club permalink
  14. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 23:08:18 JST rees rees
    in reply to
    • Moon
    • Machismo
    • NEETzsche
    @Zerglingman @NEETzsche @Moon >it's 200% meme
    use real words
    In conversation about a year ago from social.076.moe permalink
  15. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 23:06:19 JST rees rees
    in reply to
    • Moon
    • NEETzsche
    @NEETzsche @Moon doesn't take long if you are polyglot
    In conversation about a year ago from breastmilk.club permalink
  16. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 23:06:18 JST rees rees
    in reply to
    • Moon
    • NEETzsche
    @NEETzsche @Moon rust isn't a meme language, it's 10y old and has made its way into the linux kernel. you want an excuse to not learn it
    In conversation about a year ago from breastmilk.club permalink
  17. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 22:59:07 JST rees rees
    in reply to
    • Moon
    • NEETzsche
    @NEETzsche @Moon json-ld is just json, every language can handle it. if we are talking about json decoding performance then rust is a leader in it
    In conversation about a year ago from breastmilk.club permalink
  18. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 22:55:14 JST rees rees
    in reply to
    • Moon
    • NEETzsche
    @NEETzsche @Moon rust supports ruby-like string manip
    In conversation about a year ago from social.076.moe permalink
  19. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 22:55:13 JST rees rees
    in reply to
    • Moon
    • NEETzsche
    • rees
    @NEETzsche @Moon compile time so zero-cost unlike templating languages which need to do it at runtime
    In conversation about a year ago from breastmilk.club permalink
  20. rees (rees@breastmilk.club)'s status on Saturday, 09-Dec-2023 22:55:12 JST rees rees
    in reply to
    • Moon
    • NEETzsche
    @NEETzsche @Moon yeah and it's not problem for UI or web stuff, look at dioxus, full fledged UI based on CSS rendering and react-style component rendering

    https://dioxuslabs.com/
    In conversation about a year ago from breastmilk.club permalink

    Attachments

    1. Dioxus - Reliable Rust apps that run anywhere
      An elegant GUI library for Rust, inspired by React. Supports Web, Desktop, SSR, Liveview, and Mobile.
  • After
  • Before

User actions

    rees

    rees

    Tags
    • (None)
    ActivityPub
    Remote Profile

    Following 0

      Followers 0

        Groups 0

          Statistics

          User ID
          25498
          Member since
          6 May 2023
          Notices
          624
          Daily average
          1

          Feeds

          • Atom
          • Help
          • About
          • FAQ
          • TOS
          • Privacy
          • Source
          • Version
          • Contact

          ０７６萌ＳＮＳ is a social network, courtesy of ０７６. It runs on GNU social, version 2.0.2-beta0, available under the GNU Affero General Public License.

          Creative Commons Attribution 3.0 All ０７６萌ＳＮＳ content and data are available under the Creative Commons Attribution 3.0 license.